W H I T E
P A P E R
12 PROVEN CYBER
INCIDENT RESPONSE
STRATEGIES
Prepare, Respond, and
Remediate—The Winning Formula
SECURITY
REIMAGINED
12 Proven Cyber Incident
Response Strategies
Prepare, Respond, and Remediate—The Winning Formula
Executive Summary
Introduction
Cyber threats are constantly evolving, increasing
in both sophistication and how they impact
victims. In today’s world, you must battle a wide
range of adversaries, from hacktivists to statesponsored threats. The most significant of these
threats are advanced attackers who carefully plan
and execute breaches into victim organizations.
These targeted attackers quietly insert
themselves into your environment, leveraging
legitimate credentials to survey the environment
and dig in deeper for extended operations. They
can stay hidden for months—or years—all while
advancing their mission, be it stealing your data or
using your organization’s environment as a way to
infiltrate another victim.
Today’s targeted threat attackers prefer long-term
operations with an emphasis on stealth. They sneak
into your network undetected, perform prolonged
reconnaissance, and leverage legitimate credentials
that allow them to stay hidden for months or even
years. The objectives of these advanced attackers
may vary but often their mission is to steal your
most valuable data. This could be personally
identifiable information (PII), personal health data
(PHI), credit card information, corporate trade
secrets, intellectual property, or information on
mergers and acquisitions.
To battle today’s threats, you need to be
prepared. You need an effective incident response
(IR) strategy that enables you to fight back.
Specifically, you need to prepare for, respond to
and remediate security incidents.
Most importantly, your IR team needs to have
a full arsenal of experienced people, proven
techniques and methodologies, and the right
technology to analyze the attacker’s activities and
piece together a coherent story of the attack. This
story should include details of how the attacker
got into your environment, how the attacker
moved laterally from system to system, and what
data was taken.
You need an effective incident
response (IR) strategy that enables
you to fight back. Specifically, you need
to prepare for, respond to, and
remediate security incidents.
In 2014, Mandiant, a FireEye company, found
that targeted threat actors remain active in
environments for an average of 205 days1 before
being detected. Not surprisingly, they can do a
lot of damage within that time frame, and the
price tag of this damage is increasing. Over the
last two years alone, the average cost of a single
data breach has increased 23% to $3.79 million.2
What’s more, the vast majority of companies aren’t
detecting breaches themselves. In 2014, 69% of
victims were informed by a third party—such as law
enforcement—that they were breached.3
A full 66% of organizations cite skills shortage as
the main impediment to effective IR.4 As a result
of this skills shortage, 37% of organizations can’t
distinguish malicious events from nonevents.5
An effective cyber-attack response can make a
significant difference in the impact and duration of
a breach. Reducing the mean time to remediation
(MTTR)—the time between detecting a security
incident and successfully remediating it—can shrink
what could otherwise be a major news headline
into a small security incident.
Here are 12 recommended actions that will help
you prepare for and remediate security threats.
These actions will help you find attackers, shut
them out, and quickly fix the damage to minimize
MTTR.
Mandiant, a FireEye company. M-Trends 2015: A View From the Front Lines. 2015.
Ponemon Institute.“2015 Cost of Data Breach Study: Global Analysis.” May 2015.
3
Mandiant, a FireEye company. M-Trends 2015: A View From the Front Lines. 2015.
4
Alissa Torres (SANS Institute). “Maturing and Specializing: Incident Response Capabilities Needed.” August 2015.
5
Ibid.
1
2
2
12 Proven Cyber Incident
Response Strategies
Prepare, Respond, and Remediate—The Winning Formula
Figure 1:
Source: The
Cost of Malware
Containment,
Ponemon Institute,
January 2015
Extrapolated average malware alerts for
organizations participating in this study
18,000
16,937
16,000
14,000
12,000
10,000
8,000
6,000
4,000
3,218
2,000
705
Number of
alerts
Alerts deemed
reliable
Alerts
investigated
Extrapolated average values per workplace
Prepare
To give yourself the best chance of quickly
detecting and responding to incidents, you must
plan ahead. To do this, you should complete the
following five actions before you have an incident.
#1: Clearly define roles and
responsibilities
When an incident is identified, a flurry of
activity begins. Having clearly defined roles and
responsibilities for those involved in IR minimizes
confusion, prevents duplication of work, and
avoids critical gaps in the response.
The two key players in an IR are the IR team lead
and the project manager (PM). The IR team lead
heads the investigation and “owns” the technical
work associated with an IR. This is vital. It is also
critical to minimize outside influence on the IR
team lead so that the lead remains focused on
running the investigation. The IR team lead should
be allowed to follow the trail of evidence rather
than entertain theories and suggestions from
those not directly involved in the investigation.
The PM’s job is to act as a liaison between the IR
team lead and the various corporate functions to
ensure everyone works together. Although the
exact number of organizational functions involved
in IR depends on the severity of the incident, the
PM must at minimum coordinate communication
between the technical team and management. In
larger or more severe incidents, representatives
from the legal department, risk and compliance,
and corporate communications may get involved.
In smaller teams, the IR team lead and the PM can
sometimes be the same person.
3
12 Proven Cyber Incident
Response Strategies
Prepare, Respond, and Remediate—The Winning Formula
Within the IT team, it is important to designate
a person to be responsible for each major
technology area—the network, applications, the
data center, and so on. This way, the IR team can
either quickly tap into needed expertise to get
questions answered in a timely manner or—more
importantly—make infrastructure changes to
support the IR team’s efforts to contain and
remediate an incident.
One very real risk is that a department other than
the IR security team will attempt to seize control
of the investigation. During a significant breach
at one organization that Mandiant Consulting
was engaged to help, the risk and compliance
department attempted—and succeeded at—taking
ownership of the investigation. In this case, the IR
team lost the advantage of speed because it had
to go through another department that did not
fully understand the IR process and technologies
involved. This is the prime reason that roles and
responsibilities should be clearly delineated
before IR is triggered.
#2: Ensure you have full host and
network visibility
You should have a complete picture of everything
that is in your environment: specifically, endpoints
and network assets. This gives you insight into
what needs to be monitored as well as insight into
the broader implications of any security incident.
In addition, when scoping an IR, this helps the
IR team better understand the potential impact
to the environment and what it needs to do to
contain and remediate the incident.
Companies often lack complete documentation
of their networks. They also frequently don’t
have a current inventory of endpoints. That’s
because when organizations grow quickly,
accurate tracking of IT assets is not always a
priority. Furthermore, older endpoints or even
entire network segments can fall off the radar
even though they are still connected to the
network. This is especially common for companies
that focus on growth through mergers and
acquisitions. During IR, this can lead to situations
in which the IR team believes it has successfully
investigated and contained an incident, only to
realize the attacker is still active in a forgotten
network segment or endpoint.
Mandiant Consulting was once called into a large
financial services firm that had suffered a data
breach. After investigating all leads, the client
believed it had contained the incident. One week
later, Mandiant detected the attacker querying a
sensitive database again. We traced the activity to a
system in an abandoned office building that still had
power and internet connectivity. The victim
organization thought the network was no longer
connected and there were no endpoints in the
building. The reality was that the office network and
endpoints inside were still up and running and had full
connectivity back to the main environment. The
network and endpoints had simply fallen out of the
firm’s asset databases. Because the IR team lacked
full visibility, the attacker was able to re-enter the
environment through a backdoor that had been
previously placed on the system in the abandoned
office. The client ultimately had to drive three hours
to the abandoned building and unplug the system to
remove the attacker’s ability to use that backdoor
and communicate with the environment.
#3: Identify and protect your
critical assets
As you gain visibility into your hosts and networks,
you should identify where your organization’s
most valuable digital assets are stored. These
assets should be documented and kept up to
date to reflect the constant evolution of your
environment. After all, if you don’t know what
sensitive data you have or where it is located, you
won’t be able to adequately protect it.
For retailers, that means knowing the flow of
credit card information. Healthcare companies
need to know where PHI or PII is stored.
Technology companies must be aware of the
location of their source code repositories or R&D
information. For most companies, emails from
executives can also include sensitive information
on upcoming business deals. These are just a few
common examples of data that targeted threat
attackers seek to steal.
4
12 Proven Cyber Incident
Response Strategies
Prepare, Respond, and Remediate—The Winning Formula
Most companies have a very difficult time
identifying all the places that sensitive data is
stored. This can often be an issue when legacy
systems hold older sensitive data or when multiple
copies of data exist in the environment due to
poor access controls. In many targeted threat
attacks, the attackers end up knowing more
about the systems that store and process critical
information than the vast majority of your system
administrators and network architects.
Tracking your data can be as easy as creating an
Excel spreadsheet. Or you can use dedicated
asset-management tools. Either way, make sure
you know where your organization’s most valuable
data lives and ensure that the proper controls,
logging, and monitoring are in place before you
suffer an attack.
Protect User Credentials
One type of data that is commonly at risk in cyber
attacks is user credentials. Attackers will target
account credentials for various purposes. In
particular, they will target privileged accounts that
allow for easier access to systems and data in the
environment. Understanding where large sets of
user credentials are stored, such as in domain
controllers, file shares, or email servers, can help
prioritize security controls in the environment. In
some cases, attempts to dump passwords from these
systems can serve as an early warning of a targeted
cyber attack.
#4: Enhance logging, monitoring, and log
retention
The more useful information you can provide
to your security team, the better. This means
enhancing logging and monitoring in your
environment to improve your security team’s
ability to detect anomalous or malicious
activity (see “What You Should Monitor”). The
enhanced logging also provides the IR team
with additional data to help determine how the
attacker gained access and moved through your
environment.
What You Should Monitor:
The goal of logging is to provide relevant and
useful information to the security team that
enables them to rapidly identify compromised
systems and user accounts. This, in turn,
provides insight into what actions the attacker
took while in the system or with a compromised
user account. A common gap Mandiant
Consulting sees is a lack of domain name system
(DNS) and dynamic host configuration protocol
(DHCP) logging. A typical scenario is that
the security team identifies a malicious DNS
request, but without DNS logging the team has
to go through extra steps to identify the system
that sent the request and in some cases may not
QQ
Firewall logs—acceptances and denials
QQ
DNS server logs
QQ
DHCP logs
QQ
Enhanced Microsoft Windows event audit logs
QQ
External webmail access logs
QQ
Internal web proxy logs
QQ
Virtual private network (VPN) logs
QQ
Full packet captures
QQ
NetFlow network metadata
12 Proven Cyber Incident
Response Strategies
Prepare, Respond, and Remediate—The Winning Formula
have any logs to determine the actual source.
Taking the scenario a step further, if the system
were assigned an IP address through a DHCP—
very common with user workstations—the
security team might not be able to determine
the source system even when it tracks down the
originating IP address, because the IP address
can change daily.
#5: Utilize tools that allow you to do
enterprise-wide searches
Can you proactively search your entire
environment—both endpoints and network
traffic—for indicators of compromise (IOCs)?
Few companies have the tools to do this. Yet at
any given time, you need to be able to answer the
following questions:
QQ
What systems have been compromised?
QQ
What user accounts did the attacker
compromise?
QQ
What other systems did malicious activity
occur on?
Enterprise-wide searches for IOCs are critical
to properly scoping out a large breach. IOCs go
beyond just file names or the MD5 algorithm.
IOC searches should scour malware variants
(where the file name or MD5 is different
yet functionality is the same), for activity in
compromised user accounts, or for miscellaneous
attacker methodologies. Implementing a tool in
your environment that can rapidly search and
identify additional compromised systems or user
accounts during an incident is a fundamental
component of a successful investigation.
Enterprise-wide searches for indicators
of compromise are critical to properly
scoping out a large breach.
Some organizations attempt to build homegrown
search functions based on existing asset
management tools or custom scripts. Although
these searches may be partially effective, those
tools usually fail to sniff out all malicious activity
in an environment and are prone to missing
more sophisticated malware and attacker tactics.
However, whichever tool you use, your ultimate
goal is to routinely check your environment for
anomalous activity.
Respond
When you verify an attack in your environment,
you must respond with an investigation to fully
understand the potential impact of the breach.
In some cases, this investigation can be relatively
short, as when dealing with commodity malware
or common viruses. However, when dealing with
targeted threats, it is imperative to understand
how the attacker penetrated your environment
and the true extent of what the attacker accessed
or stole. The thoroughness of your investigation
directly affects the success of your remediation.
#6: Follow the evidence
The first rule of investigations is to follow the
evidence. It can be tempting to hypothesize
everything the attacker could have done. These
“rabbit holes” can lead to wasted cycles since
you are trying to prove or disprove theories that
may have no relevance to the attack. Instead of
speculating on what the attacker might have done,
look at the actual data—the forensic artifacts left
on compromised systems and log data—to see
what the attacker has actually done.
This is especially important when you perform
the investigation yourself. Because Mandiant
has investigated so many security incidents in so
many organizations under so many conditions, we
understand many of the ways an attack can play
out. Few organizations have this kind of extensive
internal expertise, especially when it comes to
more sophisticated targeted attacks. As a result,
they tend to overcomplicate, misconstrue, or
misinterpret the attack scenario.
6
12 Proven Cyber Incident
Response Strategies
Prepare, Respond, and Remediate—The Winning Formula
In one instance, the victim organization attempted
to analyze data on its own prior to bringing in
Mandiant Consulting. Limited forensic artifacts
were left on systems and log files since the breach
had occurred more than two years previously. As
a result of this limited logging, it was impossible to
see what accounts the attacker had used to move
laterally. The organization interpreted the lack of
findings as an indication that the attacker did not
procure legitimate user credentials and therefore
concluded that the impact of the attack was limited.
To be certain, they called us for a second opinion.
Based on the forensic artifacts that did remain, it
was immediately evident to us that the attacker
indeed had privileged account credentials and had
moved laterally through the environment.
Rather than focusing on what the evidence
actually showed, the victim organization spent
cycles attempting to prove that the attacker didn’t
have privileged credentials, ultimately consuming
critical time and resources. In the early stages of
an investigation, rabbit holes like this can lead to
precious days of lost time.
#7: Scope the entire incident
Properly scoping an incident is a primary focus
of any investigation. Scoping an incident involves
identifying all systems with which the attacker
has interacted. This includes systems where the
attacker has placed persistent malware, executed
utilities, harvested data, or simply logged into as
part of the reconnaissance phase. Fully scoping an
incident includes asking the following questions:
QQ
How did the attacker gain access to the
environment?
QQ
How did the attacker maintain access to the
environment?
QQ
How did the attacker move laterally
throughout the environment?
QQ
What data was stolen from the environment?
QQ
What is the impact of the breach?
QQ
Has the breach been contained?
Based on these questions, it’s clear that
identifying attacker malware used in a
compromise is only one aspect of scoping.
Attackers can access systems without placing
malware on them—a common scenario in
incidents where data is stolen.
Today, more advanced attackers rely on malware
as a means to gain an initial foothold within an
organization. After that initial foothold is established,
they shift their tactics to using legitimate means
of remote access, such as an organization’s virtual
private network (VPN). Mandiant Consulting has
observed attack groups that subsequently remove
all backdoors they initially placed in an environment
and rely exclusively on a company’s VPN to maintain
access to the environment.
In addition, attackers will use legitimate
credentials to move laterally and exfiltrate data
from the environment. In fact, an attacker can
steal all of your sensitive data without installing
any backdoors at all. In one Mandiant engagement,
the attacker had only installed six backdoors in
the victim organization’s environment, yet had
accessed more than 600 systems while dumping
passwords and searching for sensitive data.
Traditional investigative techniques may involve
running multiple antivirus products or rootkit
detection utilities, or interacting live with suspect
systems. But depending solely on traditional
techniques can cause you to miss the majority
of attacker activity, or, worse, destroy critical
evidence remaining on the system. To determine
the full extent of a compromise, IR teams must
analyze both the systems with evidence of
malware and the systems that the attacker
accessed. This includes focusing on non-malwarerelated evidence that attackers leave behind,
such as a logon to the system, files the attacker
accessed, or folders the attacker browsed.
7
12 Proven Cyber Incident
Response Strategies
Prepare, Respond, and Remediate—The Winning Formula
Common examples of attacker activity on compromised systems include:
Attacker Activity
DESCRIPTION
Reconnaissance
Attackers may scan the environment to look for specific services or operating systems. For example,
the attacker may scan for server message block (SMB) (TCP port 445) to identify other Windows
operating systems, open network shares, or remote desktop protocol (RDP) (TCP port 3389) to look for
interactive remote access to Windows systems.
Lateral movement
The attacker logs into other systems in the environment with compromised credentials for a legitimate
account. The attacker may execute utilities or browse the file system for files of interest.
Data theft
Reconnaissance and lateral movement often precede data theft. The attacker will collect, stage, and
exfiltrate data from environments. During the staging process, the attacker will typically use a single
system to harvest and compress the data. Exfiltration can occur any number of ways, including through
attacker backdoors, custom data-transfer utilities, or public data sharing websites.
Persistent backdoors
The attacker may place persistent backdoors on systems to maintain access to an environment. These
backdoors can range in functionality from a simple reverse shell to full-featured backdoors that allow
full command-line and graphical user interface (GUI) based interaction with the compromised system.
#8: Understand your attacker
Understanding the motives and the capabilities of
an attack group helps guide an investigation. For
experienced investigators, knowing the attacker’s
tools, techniques, and procedures can serve as
a shortcut to identifying additional evidence of
compromise or potential targets for data theft.
For victim organizations, understanding the
attacker’s motives and historical targets can help
determine the potential business risk.
In one Mandiant Consulting investigation, forensic
artifacts showed that the attacker exfiltrated
highly sensitive engineering data about nextgeneration technology. Through forensic analysis,
Mandiant was able to reconstruct an accurate
picture of all data that the attacker stole. Based
on that information and our knowledge of the
threat actor, the victim organization modified its
future business plans to account for increased
competition in the market space.
Often, organizations struggle to identify an
attacker due to a lack of current—or detailed—
threat intelligence. In addition, targeted threat
attackers’ tools and techniques are beginning
to merge, which makes attributing an attack to
the right threat group even harder. Over the last
few years, we have identified a new trend: cyber
criminals are stealing a page from the playbook
of APT actors, while APT actors are using tools
widely deployed by cyber criminals. As these
techniques merge, discerning attacker goals
becomes critical to gauging the impact of incidents
and building an effective IR strategy.
Given these tactical overlaps, you need to keep an
open mind when you assess attackers’ motives.
Intent cannot be determined based on one
technique or tool in isolation.
#9: Ensure proper communication
During an investigation, the security team has
many open lines of communication. Often, you’ll
need to communicate complex technical topics
to non-technical audiences in a time-sensitive
manner. In doing so, you need clear, concise, and—
above all—accurate information.
To make sure you can relay this information
effectively, here are some communications tips.
QQ
QQ
QQ
Understand when a security event
becomes a security incident. This involves
distinguishing a commodity malware
infection (an event) from a targeted attack
in which someone gains access to a critical
system or service (an incident).
Notify the IR team. The team should in turn
be conservative about whom it tells and what
it tells them based on the severity of the
incident and potential impact to the business.
In particular, avoid information overload for
executives, who can get lost in the technical
details.
Avoid premature conclusions. Mandiant
Consulting has been in situations where
internal technology teams have mistakenly
concluded that there was no evidence of data
theft, then called us in only to discover the
8
12 Proven Cyber Incident
Response Strategies
QQ
QQ
QQ
Prepare, Respond, and Remediate—The Winning Formula
true extent of the damage, which included
major data breaches. Unfortunately, by
then, executives had already gone public
with announcements that no data had
been compromised, and eventually had to
backpedal considerably. Also, information
about an incident can spread through
unofficial corporate channels and get
distorted along the way. If you’re not careful,
IR communication can turn into a bad game of
telephone. To prevent this, provide accurate
and concise information on an as-needed
basis that doesn’t sound alarm bells when
no alarm is necessary and doesn’t cause the
organization to communicate erroneous
positive information to the external world.
Remediate
Keep both internal and external briefings
to a minimum. It’s very common, especially
in the early days of an attack, for executives to
demand hourly updates. This means that the IR
team spends more time prepping for briefings
than analyzing the logs and other data. Balance
the need for communication with the need to
move the investigation forward.
#10: Contain and eradicate the threat
Don’t get sidetracked by speculation.
In many cases, organizations have to
scramble to stay ahead of the narrative that
builds independently of the investigation.
Don’t let public speculation about how
an attacker gained entry put pressure on
investigators to disprove theories. These
rabbit holes distract rather than help an
investigation. The IR teams needs to focus
on scoping and containing the incident.
Investigations can take weeks or months,
and the facts emerge over time.
Consider your legal reporting
requirements. If regulated data, such as PII,
PHI, or payment card information is stolen,
you may have a legal obligation to report it.
In addition, many organizations may face
contractual obligations with other companies
or regulatory bodies.
An effective IR consists of two parts: an
investigation and a remediation. The investigation
drives what remediation activities are necessary.
Remediation consists of the following:
1.
Contain the attack: Disrupt the attacker’s
ability to operate in the environment.
2.
Eradicate the attacker from the environment:
Remove the attacker from the environment
and prevent the attacker from easily
accessing the environment again.
3.
Implement long-term strategic changes to the
environment: Enhance the security posture
of the organization through long-term goals.
Containing an attack means stopping the bleeding.
Containment is meant to disrupt the attacker’s
ability to operate in an environment. Activities
at this stage can include resetting compromised
credentials, blocking known malicious IP
addresses or domains, or removing infected
systems from an environment.
Eradicating an attacker from the environment
means kicking attackers out and keeping them
from coming back in. This can include fixing a
vulnerability that may have led to the attack,
resetting all passwords in an environment, or
implementing two-factor authentication.
In addition, if the victim organization had previously
not performed adequate logging and monitoring,
this is the appropriate time to enhance that
throughout the organization. The additional
visibility can help determine whether an attacker is
trying to regain access to the environment.
#11: Ensure your remediation actions
are working
Remediation events have a lot of moving parts.
You should verify that all your remediation action
items have been implemented and test each one
to validate that they are working as expected.
During a remediation, Mandiant Consulting will
9
12 Proven Cyber Incident
Response Strategies
Prepare, Respond, and Remediate—The Winning Formula
You should verify that all your remediation action items have
been implemented and test each one to validate that they are
working as expected.
frequently deploy additional monitoring for weeks
or months to see if the attacker returns. Some
organizations will even put “red teams” in place to
simulate attacks and assess the effectiveness of
the new controls.
#12: Set long-term strategic
recommendations
After making sure that the threat is contained
and eradicated, you need to establish and begin
implementation of strategic recommendations.
These longer-term actions will enhance your
organization’s security posture overall by
helping to mitigate the impact of future attacks,
inhibit an attacker’s ability to operate in the
environment, and improve your ability to detect
and respond to threats.
Long-term strategic initiatives can include:
•
Enhancing network segmentation to restrict
lateral movement
•
Enhancing logging and monitoring to better
detect anomalous activity
•
Enhancing protection for privileged user
accounts to minimize misuse
•
Creating an investigation-ready environment
to quickly scope incidents
•
Investing in more security professionals
and working to advance their experience
and skill sets
Prepare, Respond, Remediate—
The Winning Formula
As attackers become more sophisticated, so must
your detection and IR capabilities. Implementing
the 12 recommended actions will better position
your organization to effectively prepare for,
respond to, and remediate security incidents. An
effective and mature IR program continues to
adapt its tactics and techniques based on lessons
learned from every incident and the evolving
threat landscape.
ABOUT FIREEYE
FireEye protects the most valuable assets in the world from those who have them in their sights. Our combination of
technology, intelligence, and expertise—reinforced with the most aggressive incident response team—helps eliminate the
impact of security breaches. We find and stop attackers at every stage of an incursion. With FireEye, you’ll detect attacks as
they happen. You’ll understand the risk these attacks pose to your most valued assets. And you’ll have the resources to
quickly respond and resolve security incidents. FireEye has over 3,100 customers across 67 countries, including over 200
of the Fortune 500.
10
To learn more about Mandiant Consulting services visit
www.fireeye.com/services
FireEye, Inc. | 1440 McCarthy Blvd. Milpitas, CA 95035 | 408.321.6300 | 877.FIREEYE (347.3393) | [email protected] | www.fireeye.com
© 2015 FireEye, Inc. All rights reserved. FireEye is a registered trademark of
FireEye, Inc. All other brands, products, or service names are or may be trademarks
or service marks of their respective owners. WP.CIRS.EN-US.112015