Are CAATs keeping you awake at night?
SUMMARY:
 The importance of using Computer-Assisted
Audit Techniques is discussed.
 A challenge is made regarding the audit
profession’s traditional methodology.
 The benefits of CAATs to both management and
audit.
We would also point out that CAATs are not solely the
domain of the auditor. CAATS or the concept of them
can also be used as management tools. They can exist
for management and be specified by management to
assist them in their on-going tasks and responsibilities.
CAAT capability need to be considered up front when
systems and applications are being considered for
operating models, they should be implemented with MI
and control in mind to reduce errors, identify outliers
and rectify.
We’ve probably all suffered an interrupted night’s sleep
due to our feline friends! There is however another
variety of CAATs – Computer-Assisted Audit Techniques
– that may be causing those in the audit profession
along with the Management and Audit Committees they
serve, the odd sleepless night. If you are wondering
about the last time you considered using a CAAT
and/or deployed one, or perhaps you have no idea as
to what a CAAT is, read on.
Traditional auditing vs. CAATs
The traditional method of auditing allows auditors to
build conclusions based upon a limited sample of a
population, rather than an examination of all available
– or a large sample of – data. Management realises
that they conduct thousands or perhaps millions of
transactions a year and the auditor only sampled a
handful. The auditor will then state that they conducted
the sample based upon Generally Accepted Audit
Standards (eg GAAS) and that their sample was
statistically valid.
Another common criticism of the audit profession
occurs after a problem emerges. Management might
ask, “Where were the auditors?” This is a futile
question, because nobody can see beyond the present.
© 2013 Kingston Smith Consulting LLP
Big CAATs!
CAATs, is a methodology of analysing large volumes of
data looking for anomalies. A well designed CAAT audit
will not review a sample, but rather a complete review
of all transactions. Using CAATs, the auditor will extract
every transaction the business unit performed during
the period under review. The auditor will then perform
tests to determine if there are any problems in the
data. The use of CAATs as part of continuous
monitoring (if objective and purpose of use is clear)
22/04/2013
also allows for continuous assurance that enables
correct action to be take more quickly and enables an
assurance report to be made available in real time
which is particularly useful on key controls. This in
effect enables IA to monitor business systems and their
procedures, activities, transactions and events in a
real-time manner.
Why, if internal and external auditors are effectively
deploying CAATs, are we still seeing so many errors,
data breaches, vulnerabilities etc? The answer is
perhaps in the detail.
Specialised software
CAATs allow auditors to test for specific risks. Typical
examples include analysing for duplicate vendor
payments; combining data from the payroll with the
purchase leger to identify employees directing
payments to themselves; looking for inappropriate
access to seldom-used accounts; identifying insurance
claims paid after a policy lapsed; etc. The possibilities
are endless. None of these tasks would be feasible by
use of manual techniques.
In the most general terms, CAATs can refer to any
computer program utilised to improve the audit
process. Generally, however, it is used to refer to data
extraction and analysis software. This would include
programs such as spreadsheets (eg Excel), Databases
(eg Access), statistical analysis (eg SAS) and business
intelligence (eg Crystal Reports or Business Objects).
There are companies that have developed dedicated
specialized data analytic software specifically for
auditors. Examples include Audit Command Language
(ACL), Quick Data Analyse and Conversion (QDAC), and
Interactive Data Extraction and Analysis (IDEA) etc.
Traditional audit vs. CAATs on specific
risks
Which looks better in an audit report:
“Audit reviewed 50 transactions and noted one
transaction that was processed incorrectly”
or
“Audit used computer-assisted audit techniques and
tested every transaction over the past year. We noted
XXX exceptions wherein the company paid YYY amount
incorrectly.”
What’s new, pussycat?
However, the CAAT-driven review is limited only to the
data saved on files in accordance with a systematic
pattern. Much data is never organised in this way. In
addition, saved data often contains deficiencies, is
poorly classified, is not easy to retrieve, and may have
integrity issues. In certain audits CAATs can't be used
at all. But there are also audits which simply can't be
made with due care and efficiency without CAATs.
The purists amongst you are probably thinking, “We’ve
seen and heard this all before and there is plenty of
guidance on the subject matter.”
A leopard cannot change its spots!
Arguably the most comprehensive guidance resides
within ISACA® (Information Systems Audit and Control
Association) “Use of Computer-Assisted Audit
Techniques”. The ISACA publication is excellent: clear,
concise and to the point. That said, there remains a key
underlying question.
“The specialised nature of information systems (IS)
auditing and the skills necessary to perform such
audits require standards that apply specifically to IS
auditing. One of the goals of ISACA is to advance
globally applicable standards to meet its vision. The
development and dissemination of the IS Auditing
© 2013 Kingston Smith Consulting LLP
ISACA states that:
22/04/2013
Standards are a cornerstone of the ISACA professional
contribution to the audit community”.
“CAATs provide auditors with tools that can identify
unexpected or unexplained patterns in data that may
indicate fraud. Whether the CAAT is simple or complex,
data analysis provides many benefits in the prevention
and detection of fraud.”
For auditors to ensure a comprehensive approach to
acquire, analyse, and report on business data, they
must make certain the organization continuously
monitors user activity on all computer systems,
business transactions and processes, and application
controls. The Institute of Internal Auditors recently
published a GTAG (Global Technology Audit Guide) on
Continuous Monitoring.”
sufficient, reliable and relevant evidence to achieve the
audit objectives. The audit findings and conclusions
are to be supported by the appropriate analysis and
interpretation of this evidence.”
Standard S7 Reporting states: “The IS auditor should
have sufficient and appropriate audit evidence to
support the results reported.”
Standard S14 Audit Evidence states: “The IS auditor
should obtain sufficient and appropriate audit evidence
to draw reasonable conclusions on which to base the
audit results.”
So what’s the problem?
At first glance there doesn’t appear to be one. ISACA
states:
As entities increase the use of information systems to
record, transact and process data, the need for the IS
auditor to utilise IS tools to adequately assess risk
becomes an integral part of audit coverage. The use of
computer-assisted audit techniques (CAATs) serves as
an important tool for the IS auditor to evaluate the
control environment in an efficient and effective
manner. The use of CAATs can lead to increased audit
coverage, more thorough and consistent analysis of
data, and reduction in risk.
CAATs include many types of tools and techniques,
such as generalised audit software, customised
queries or scripts, utility software, software tracing and
mapping, and audit expert systems.
CAATs may be used in performing various audit
procedures including:
Standard S3 Professional Ethics and Standards states:
“The IS auditor should exercise due professional care,
including observance of applicable professional
auditing standards.”
Standard S5 Planning states: “The IS auditor should
plan the information systems audit coverage to
address the audit objectives and to comply with
applicable laws and professional auditing standards.”

Tests of details of transactions and balances.

Analytical review procedures.

Compliance tests of IS general controls.

Compliance tests of IS application controls.

Penetration testing.
Standard S6 Performance of Audit Work states: “During
the course of the audit, the IS auditor should obtain
© 2013 Kingston Smith Consulting LLP
22/04/2013
Are you allergic to CAATs?
Auditors need to take a step back and rethink their
approach. IT audit has long been considered by the
profession to be a specialist discipline. Perhaps these
specialists have been taken too much for granted and
the more “generalist” approach has taken precedence.
The multifaceted “hybrid auditor” is becoming more
and more popular; unfortunately, as with most hybrids,
they are prone to failure!
Are IT Auditors actually aware of all of the tools they
have available in their armoury? The answer is
probably not. This hardly comes as a surprise, given
that what appears on a weekly basis, there are new
solutions available to facilitate continuous monitoring,
active defences etc.
Do IT Auditors truly understand all of the respective
facets of these solutions? Are they able to devise
effective programmes of work to test them? The stark
reality is that the technology is moving far quicker than
the auditors, who are struggling to keep pace.
A recent review of Active Directory audit(s)
demonstrated that there was a lack of consistency in
terms of approach, controls to be considered,
recommendations, remedial action etc. The elapsed
time to perform these audits in like-for-like
environments varied between two and eight weeks,
which suggests that there is something seriously awry
in terms of the approaches. Having reviewed the
results, of real concern was the absence of any
automation being applied by the auditors and that the
findings were on the whole somewhat rudimentary. In
fact there are automated tools that, if deployed
effectively, mean this type of audit is a relatively
straightforward and standard undertaking.
Clients we speak to are often surprised when we show
them exactly how many spreadsheets are in use in
their organization. They are even more surprised when
we demonstrate the nature and extent of the risk to
their organization of relying on the flawed, erroneous or
mistaken data contained in the spreadsheets. Senior
management and risk functions often assume that the
IT department has grasped this risk and has mitigated
© 2013 Kingston Smith Consulting LLP
it – but this is not usually the case. There are
automated tools which can highlight potential sources
of error in individual critical spreadsheets as well as
provide overall governance – including inventory,
change control, etc. but how frequently are these
deployed? In our experience rarely if not at all; which is
surprising given that if you are subject to regulation or
legislation such as the Data Protection Act, the
Financial Services Regulators, Basel II/III, Sarbanes
Oxley, Payment Card Industry Data Security Standard
etc, you may in fact be obliged to implement proper
controls.
It could be argued that the use of such tools is moving
away from that for which CAATs were traditionally
envisaged. That said, if the tools are there why not use
them? This could be down to a combination of many
factors, for example no knowledge of the product, not
qualified to deploy the product, procurement budget
etc. Unfortunately, the reality is that auditors are falling
behind the technology and the number of issues failing
to be identified is on the rise as, correspondingly, are
the number of data breaches etc.
Clearly there remains a significant number of specialist
IT Auditors, but are they losing their way? Has internal
audit lost its way in general? That’s probably too broad
a subject for us to tackle here, so let’s try and address
the issue at hand: what is the decision factor for using
CAATs? And what type of CAATs should you be using?
Decision factors for using CAATs
When planning the audit, the IS auditor should
consider an appropriate combination of manual
techniques and CAATs. In determining whether to use
CAATs, the factors to be considered include:

Relevant knowledge, expertise, and experience of
the IS auditor

Availability of suitable CAATs and IS facilities

Efficiency and effectiveness of using CAATs over
manual techniques

Time constraints
22/04/2013


Integrity of the information system and IT
environment
Level of audit risk
CAATs always land on their feet
Claw enforcement
Where CAATs are used to extracting information for
data analysis, the IS auditor should verify the integrity
of the information system and IT environment from
which the data are extracted.
The major steps to be undertaken by the IS auditor in
preparing for the application of the selected CAATs
include the following:

Set the audit objectives of the CAATs, which may be
included in the terms of reference for the exercise.

Determine the accessibility and availability of the
organisation’s IS facilities, programs/systems and
data.

Clearly understand the composition of the data to
be processed including quantity, type, format and
layout.

Define the procedures to be undertaken (eg,
statistical sampling, recalculation, confirmation).

Define output requirements.

Determine resource requirements (personnel,
CAATs, processing environment – the
organisation’s IS facilities or audit IS facilities).

Obtain access to the organisation’s IS facilities,
programs/systems and data, including file
definitions.

Document CAATs to be used, including objectives,
high-level flowcharts and run instructions.
You don’t have nine lives!
It is critical that the IS auditor obtain reasonable
assurance of the integrity, reliability, usefulness and
security of the CAATs through appropriate planning,
design, testing, processing and review of
documentation. This should be done before reliance is
placed on CAATs, and after they are changed. The
nature, timing and extent of testing is dependent on
the commercial availability and stability of the tools.
Custom CAATs should receive additional review and
testing to ensure they are operating as expected.
© 2013 Kingston Smith Consulting LLP
CAATs can be used to extract sensitive
program/system information and production data that
should be kept confidential. The IS auditor should
clearly understand company data classification and
data handling policies to properly safeguard the
program/system information and production data with
an appropriate level of confidentiality and security. In
doing so, the IS auditor should consider the level of
confidentiality and security required by the organisation
owning the data and any relevant legislation (eg Data
Protection), and should consult others, such as legal
counsel and management, as necessary.
The IS auditor should use and document the results of
appropriate procedures to provide for the ongoing
integrity, reliability, usefulness and security of the
CAATs. For example, this should include a review of
program maintenance and program change controls
over embedded audit software to determine that only
authorised changes have been made to the CAATs.
22/04/2013
When CAATs reside in an environment not under the
control of the IS auditor, an appropriate level of control
should be in effect to identify changes to the CAATs.
that this approach generates significantly enhanced
results on our engagements as our team bring in-depth
knowledge of all facets of IT audits and operations,
leading to higher quality insights.
Qualified and credible
Our professionals all hold appropriate qualifications
and are extremely familiar with relevant best practices.
KSC is a firm believer in staff development and each
professional receives a minimum of 40 hours technical
training per annum to ensure they remain up to date. In
addition to their wide experience, this adds to their
credibility in dealing with your staff at all levels up to
and including the executive, as well as presenting in
context to regulators.
Purrfect!
How can we help you?
KSC only deploys experienced and informed
consultants to work with you – at a more affordable
price. All of our consultants have had significant
exposure to real-world commercial IT risks as
practitioners and in industry roles. Our experience
model guarantees all members of our team have at
least 10 years relevant experience. We strongly believe
To learn more about how KSC can assist in fulfilling
your audit needs, please feel free to contact Mark Child
– Partner.
Tel:
+44 (0)20 7566 3731
Email: [email protected]
About Kingston Smith Consulting LLP
Kingston Smith Consulting (KSC) is the specialist consulting practice of the top 20 accountancy firm Kingston Smith LLP.
Established in 2009, KSC provides services in all aspects of Technology Risk Management, Governance and Controls
Assurance and Legal and Regulatory Compliance. In addition, we have a team skilled at specialist services such as due
diligence, supplier selection and third party management. We maintain strong relationships with allied service providers in
order to be your “one stop” consulting solution.
Kingston Smith Consulting LLP
Devonshire House, 60 Goswell Road, London EC1M 7AD, UK Telephone +44 (0)20 7566 3732 Fax +44 (0)20 7566 4010
[email protected] www.kscllp.co.uk
A list of partners is available for inspection at the above address.
Registered in England and Wales as a Limited Liability Partnership: No OC341786 Registered office: Devonshire House, 60 Goswell Road, London EC1M 7AD, UK
© 2013 Kingston Smith Consulting LLP
22/04/2013