Interested in learning
more about security?
SANS Institute
InfoSec Reading Room
This paper is from the SANS Institute Reading Room site. Reposting is not permitted without express written permission.
How to Choose a Qualified Security Assessor
Copyright SANS Institute
Author Retains Full Rights
Sponsored by SecureWorks
How to Choose a
Qualified Security Assessor
A SANS Whitepaper – November 2010
Written by Dave Shackleford
QSA Guidelines: What’s New?
What’s Still Broken:
You Get What You Pay For
Choosing a QSA
QSA Guidelines: What’s New?
Since its inception in 2004, the Payment Card Industry Data Security Standard (PCI DSS) has
required financial service providers and large merchants to use Qualified Security Assessors
(QSAs) to conduct onsite assessments and audits of security and compliance controls. More
recently, the PCI DSS standard has expanded to include training guidelines for QSAs and other
improvements. However, QSAs—and the services they render—still vary widely. Among assessors, there are vast differences in methodologies, thoroughness, technical skills, and many other
areas. In other words, the outcome of an assessment is only as good as the qualified assessor.
Now, PCI DSS v2.0, released October 30, contains numerous clarifications and additional areas
of guidance for assessments. For example, the new standard specifies that the first step of any
PCI DSS review is to carefully define the scope of the assessment, itself, by identifying all flows
and locations of cardholder data within an environment. Because many organizations do not
know every location where cardholder data may be located in their systems, a QSA must be
able to understand network architecture, application data handling, operating system security,
storage and database technology, and many other IT and business functions in order to make
those assessments.
Another new area of guidance in PCI DSS v2.0 is its allowance of virtualization technologies
and how to assess them. As more organizations look to realize cost savings and efficiencies by
implementing server and application virtualization, QSAs need to understand how this technology differs from traditional client/server technologies they’re used to assessing.
With virtualization capabilities, numerous server “instances” can be created and run from a
single physical system, and many QSAs have considered this to be noncompliant in the past.
Section 2.2.1 of PCI v2.0 allows virtualization to be used, but specifies that each virtual server
can run only one critical function (for example, one virtual machine can run web services, while
another would run database services). So QSAs will need to understand virtual network segmentation, virtualization-specific controls, and surrounding physical IT controls that interface
with the virtualization platforms.
SANS Analyst Program
1
How to Choose a Qualified Security Assessor
What’s Still Broken: You Get What You Pay For
One common concern that many merchants and service providers have when evaluating QSA
firms is the cost of the assessment. For many, keeping costs as low as possible is a priority, and
the lowest-priced QSA firm will win the business. This is a dangerous strategy. Many find out
later that they got the assessment they paid for.
For example, be wary of one major cost-cutting tactic that many QSA firms employ: Too little
onsite evaluation time. This can be a problem because much of the control validation must be
done in person, and QSAs may miss important details by reviewing controls remotely. Although
it will vary significantly for a large enterprise organization, a week or more of onsite time may
be necessary for proper validation at the main offices, with one or two days allotted for each
additional separate location. There are legitimate ways to reduce overall time spent onsite,
such as sampling similar environments and validated systems using the same controls, rather
than testing each individual system in the group.
Another area where you get what you pay for is in QSA technical and firm experience. Without
a sound technical background and understanding of where cardholder data flows, a QSA could
potentially over- or under-scope a PCI assessment, which may result in project inefficiency at
best, or at worst, could result in major exposure and compliance violations. There are other
concerns around QSAs prescribing expensive solutions to inexpensive problems, even as they
miss the real problems that need to be repaired.
So, organizations must examine the QSAs experience against their structure and needs. Some
QSA firms have limited IT security depth among their staff; so, organizations should do a thorough assessment of how much depth of background they require and hire accordingly.
Finally, some QSAs have reputations for providing “rubber stamp” compliance—in other words,
being extremely lenient in evaluating PCI DSS controls and charging less for their quickie
assessments.
Some organizations, thinking only of assessment costs, will undoubtedly select a firm that is
inexpensive, does most of the work remotely, and seeks only to help the company “check the
checkboxes.” However, in so doing, they’re putting their organizations at risk of compliance violations and increased overall costs—defeating the purpose of the “low
cost” assessment to begin with. They’re also missing a valuable opportunity to genuinely improve security where needed.
SANS Analyst Program
2
How to Choose a Qualified Security Assessor
Choosing a QSA
Organizations can save themselves significant expense and trouble by knowing as much as
they can about their environments and cardholder data before going into negotiations with
an unknown QSA. For example, if you know your organization is processing card data through
virtual systems, the QSA would need the right skills in virtualization as well as the applications
and systems processing and touching the card data. With knowledge of their environments,
acquiring organizations will also better be able to determine if an assessment proposal is overor under-scoped.
Knowing your environment also helps when developing selection criteria for the QSA. Although
each individual organization’s criteria will vary widely, answers to these five questions apply to
all QSAs under consideration:
1. For what types of organizations have you performed PCI DSS assessments?
How many assessments have you performed and when?
This information should be on the resume of the QSA and its firm. The type of clients
they’ve serviced is relevant if they’re in the same sector as your organization. Companies in the same vertical (retail, finance, etc.) use similar payment card processing equipment and applications and will probably have related use cases and data
flows. Leveraging an assessor’s previous experience with the same type of business,
technologies, architectures, and compliance rules will expedite the assessment and
raise the level of in-depth security guidance provided.
2. What is your background?
For organizations looking to use the PCI DSS controls to improve their overall information security posture, choosing a QSA firm with deep technical knowledge and
extensive experience in information security consulting and implementation is key.
Such QSA firms will tend to be more expensive on average because they have deep
knowledge of the 12 PCI DSS areas and where they apply in complex organizations.
With these firms, you get what you pay for, as well. The better QSA firms are the ones
that go beyond the checklist and guide their clients into a comprehensive, riskbased approach to remediation and improved security.
SANS Analyst Program
3
How to Choose a Qualified Security Assessor
3. Who will be performing the work?
A very common experience for many PCI DSS assessment customers is having indepth discussions with QSAs during the data gathering and analysis phase via conference calls, meeting rooms and email. After all that, the assessors who arrive onsite
are not the same ones who worked diligently with the client to gather data and
evaluate the existing documentation, architecture and policies of the client organization. This can lead to a severe disconnect: The new QSAs sent to the site may not
grasp the nuances of the business model, controls in place, compensating controls,
and other important elements that were digested by the QSA who did the planning. Understand your QSA staffing model before signing. If the assessment is being
handled by multiple QSA consultants with varied responsibilities, investigate how
they plan to avoid disconnects and their potential impacts.
4. How do you validate and assess compensating controls?
Compensating controls are common, particularly in environments with legacy computing systems and applications, or systems that are restricted by bandwidth and
other issues. Compensating controls are put in place due to an inability to implement controls that meet the specific language of the PCI DSS requirements. Compensating controls should, however, accomplish the same goals as the intended
controls and be sufficient to protect payment card data just as the intended PCI
DSS control(s) would. An example might be a kiosk system that can’t support an
antivirus software agent due to resource consumption. In such a case, whitelisting
could be implemented as a compensating control.
Many QSAs do not adequately evaluate compensating controls. A QSA should have
a strong technical background with experience performing technical risk assessments and determining the efficacy of security controls in a variety of circumstances.
If possible, ask the QSA firm for a sample of compensating controls documentation
(sanitized) or, at a minimum, ask them to provide details on the types of compensating controls they have evaluated and approved.
SANS Analyst Program
4
How to Choose a Qualified Security Assessor
5. Are there examples of your assessments being used to improve security for
clients?
PCI DSS can be leveraged as an excellent baseline for an information security program, and should be viewed as an opportunity to improve information security as
a whole. By working diligently to meet or exceed all requirements of the PCI DSS,
organizations can measurably improve their overall security posture with the help
of knowledgeable QSAs.
In addition to checking resumes and references and questioning the assessors, be sure to
research QSAs you are considering hiring on the web. For example, a few good keyword searches
might answer a most critical question: Have any of the assessor’s clients been breached after
passing assessments by that QSA or that QSA’s firm?
SANS Analyst Program
5
How to Choose a Qualified Security Assessor
Conclusion
The selection of a QSA may be the start of a long-term relationship. Companies should look for
QSA firms that have experience with the same or similar technology to be audited. In preparation for hiring a QSA, organizations should be gathering business requirements, developing in-depth interviews and questions related to experience, and allocating time for planning
and onsite audit. Ensure that the individual QSAs you speak and work with remotely for data
gathering and assessment planning are the same ones who actually come onsite for controls
evaluation. Because compensating controls are common, look for a QSA firm with experience
in evaluating the effectiveness of your specific compensating controls.
Overall, the selection of a QSA firm will have a long and lasting effect on security and compliance for companies subject to PCI DSS compliance. Making the right choice can ensure beneficial outcomes for both meeting PCI DSS compliance and improving security posture in the
long run.
SANS Analyst Program
6
How to Choose a Qualified Security Assessor
About the Author
Senior SANS Analyst Dave Shackleford is director of security assessments and risk & compliance at Sword & Shield Enterprise Security, a SANS instructor and GIAC technical director. He
has consulted with hundreds of organizations in the areas of regulatory compliance, security,
and network architecture and engineering. He has worked as chief security officer for Configuresoft, chief technology officer for the Center for Internet Security, and as a security architect,
analyst, and manager for several Fortune 500 companies.
SANS Analyst Program
7
How to Choose a Qualified Security Assessor
SANS would like to thank this paper’s sponsor
SANS Analyst Program
8
How to Choose a Qualified Security Assessor
Last Updated: July 13th, 2017
Upcoming SANS Training
Click Here for a full list of all Upcoming SANS Events by Location
Security Awareness Summit & Training 2017
Nashville, TNUS
Jul 31, 2017 - Aug 09, 2017
Live Event
SANS San Antonio 2017
San Antonio, TXUS
Aug 06, 2017 - Aug 11, 2017
Live Event
SANS Hyderabad 2017
Hyderabad, IN
Aug 07, 2017 - Aug 12, 2017
Live Event
SANS Boston 2017
Boston, MAUS
Aug 07, 2017 - Aug 12, 2017
Live Event
SANS Prague 2017
Prague, CZ
Aug 07, 2017 - Aug 12, 2017
Live Event
SANS New York City 2017
New York City, NYUS
Aug 14, 2017 - Aug 19, 2017
Live Event
SANS Salt Lake City 2017
Salt Lake City, UTUS
Aug 14, 2017 - Aug 19, 2017
Live Event
SANS Virginia Beach 2017
Virginia Beach, VAUS
Aug 21, 2017 - Sep 01, 2017
Live Event
SANS Adelaide 2017
Adelaide, AU
Aug 21, 2017 - Aug 26, 2017
Live Event
SANS Chicago 2017
Chicago, ILUS
Aug 21, 2017 - Aug 26, 2017
Live Event
SANS San Francisco Fall 2017
San Francisco, CAUS
Sep 05, 2017 - Sep 10, 2017
Live Event
SANS Tampa - Clearwater 2017
Clearwater, FLUS
Sep 05, 2017 - Sep 10, 2017
Live Event
SANS Network Security 2017
Las Vegas, NVUS
Sep 10, 2017 - Sep 17, 2017
Live Event
SANS Dublin 2017
Dublin, IE
Sep 11, 2017 - Sep 16, 2017
Live Event
Data Breach Summit & Training
Chicago, ILUS
Sep 25, 2017 - Oct 02, 2017
Live Event
Rocky Mountain Fall 2017
Denver, COUS
Sep 25, 2017 - Sep 30, 2017
Live Event
SANS SEC504 at Cyber Security Week 2017
The Hague, NL
Sep 25, 2017 - Sep 30, 2017
Live Event
SANS Baltimore Fall 2017
Baltimore, MDUS
Sep 25, 2017 - Sep 30, 2017
Live Event
SANS London September 2017
London, GB
Sep 25, 2017 - Sep 30, 2017
Live Event
SANS Copenhagen 2017
Copenhagen, DK
Sep 25, 2017 - Sep 30, 2017
Live Event
SANS DFIR Prague 2017
Prague, CZ
Oct 02, 2017 - Oct 08, 2017
Live Event
SANS Oslo Autumn 2017
Oslo, NO
Oct 02, 2017 - Oct 07, 2017
Live Event
SANS Phoenix-Mesa 2017
Mesa, AZUS
Oct 09, 2017 - Oct 14, 2017
Live Event
SANS October Singapore 2017
Singapore, SG
Oct 09, 2017 - Oct 28, 2017
Live Event
SANS AUD507 (GSNA) @ Canberra 2017
Canberra, AU
Oct 09, 2017 - Oct 14, 2017
Live Event
Secure DevOps Summit & Training
Denver, COUS
Oct 10, 2017 - Oct 17, 2017
Live Event
SANSFIRE 2017
OnlineDCUS
Jul 22, 2017 - Jul 29, 2017
Live Event
SANS OnDemand
Books & MP3s OnlyUS
Anytime
Self Paced