1
A Needs Assessment of the LETC’s Internal Self-Assessment
James Corbett, Dyan Ellinger, Savannah McEntarffer, and Alan Wheaton
OPWL 529 Needs Assessment
Boise State University
Fall 2013
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
2
Table of Contents
Executive Summary……………………………………………………………………..3
Introduction……………………………………………………………………………...5
Define Needs…………………………………………………………………………….6
Interventions…………………………………………………………………………….13
Recommendations………………………………………………………………………19
References………………………………………………………………………………20
Appendix A: Supporting Data…………………………………………………………..22
Appendix B: Performance Technology Frameworks…………………………………...23
Appendix C: Lessons Learned…………………………………………………………..30
Appendix D: Triangulation Data………………………………………………………..31
Appendix E: Interview Questions……………………………………………………….34
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
3
Executive Summary
Purpose
A needs assessment was conducted to evaluate the ABC Law Enforcement Training
Corporation’s (LETC) internal self-assessment process in preparation for federal accreditation and to
provide the CEO with recommendations to improve the quality of the internal assessments.

Current state of performance: 60% to 75% of the training accreditation standards are NOT met
during the internal self-assessment.

Desired state of performance: 100% of the training accreditation standards are met during the
internal self-assessment.
Objectives
Based on the purpose and goals of the needs assessment, four objectives were established for the
needs assessment.
1. Given the available data, define the performance gap related to the internal self-assessment results
in preparation for federal accreditation.
2. Using systematic tools, identify causes of the performance gap through analysis of evidence
based data.
3. Once the causes have been identified, develop interventions that address the identified causal
factors.
4. Using the determined interventions and performance tools, prioritize the interventions to ensure
effective and efficient recommendations are made.
Process and Findings
After identifying the actual and desired state of performance, the causes and potential
interventions were sought. The process utilized several HPT frameworks followed by rigorous data
collection which included several interviews with key stakeholders, numerous observations by the client
liaison, task analysis and extensive document review of emails, past assessments, and various doctrinal
instructions for the LETC organization.
Using the frameworks and data collected, the team identified the following primary deficiencies
affecting desired performance:



Lack of clear processes for internal assessments
Lack of constructive feedback to improve the overall process
Lack of a viable tool to manage the entire process
These are the general causes of poor performance; however, these causes of poor performance
and others were identified during the team’s assessment and are discussed in greater detail in the final
report. The HPT frameworks and the analysis combined with the data collected identified an operational
level performance issue with the potential for systemic effects at the tactical and strategic levels of the
organization. The fishbone diagram included below represents this examination, utilized with Thomas
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
4
Gilbert’s Behavior Engineering Model and its categories of data, instruments, incentives, knowledge,
capacity, and motives.
Recommendations
Following the identification of the causes of the performance deficiency, the team was able to
identify several performance interventions that would remove the barriers to desired performance. Using a
multicriteria analysis to compare these interventions based on expected results, feasibility, cost, ability to
accomplish the desired outcomes, and acceptability, the team recommends that ABC LETC consider the
following performance interventions.
1) Checklists for accreditation preparation processes
2) Tools for tracking accreditation processes
3) SharePoint Training
4) Process and Communication Flow Chart
5) Updates to LETC Organization and Regulations Manual
6) Dispute Resolution Process
While these six interventions were chosen based on their potential for a significant impact on
closing the performance gap, if resources are available, the LETC should also consider implementing
some of the remaining interventions that were identified. The analysis team recommends the organization
develop a comprehensive plan to implement the alternative sets of interventions and continuously monitor
and evaluate the performance of the internal self-assessment.
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
5
Introduction
Client Organization
The ABC Law Enforcement Training Corporation (LETC) is the client organization of this needs
assessment. The LETC was established in 2004 to prepare ABC personnel to perform as law enforcement
officers. The LETC enhances the law enforcement skills of students from other federal, state and local
agencies, as well as the international community. This is accomplished through the delivery of high
quality training that provides the knowledge and skills necessary for graduates to perform in a safe, legal
and professional manner. The LETC also develops, maintains and makes readily available up-to-date
training materials that support the standardization and professionalism of the LETC’s entire law
enforcement training system. The LETC and other law enforcement agencies also obtain accreditation for
some courses, through their accrediting body, BOSS.
Reporting
The needs assessment team reported to John Smith, Chief Executive Officer (CEO), ABC Law
Enforcement Training Corporation.
Team & Other Stakeholders
The LETC members that participated in the needs assessment are Ken Jones, Accreditation
Manager (Training Division), Bill Durango, Assistant Accreditation Manager (Training Division), Lee
Clayton, Instructional Designer (BOSS Assessor/Performance Systems Branch), Ken Crooker, Training
Specialist (BOSS Assessor/Performance Systems Branch), and Tom Brady, Training Officer.
The needs assessment team included graduate students from Boise State University: James
Corbett, Dyan Ellinger, Savannah McEntarffer, and Alan Wheaton. While Mr. Wheaton was the client
liaison and member of the LETC, all four team members conducted the research, contributed to project
deliverables, tracked the project schedule, collected and analyzed data, and developed potential
interventions related to the findings. The project was conducted remotely because all team members are
located in different states.
Purpose
A needs assessment drives thoughtful organizational decisions. Starting with identification of a
performance gap and the desired performance, a needs assessment can evaluate causes of a gap, identify
and analyze data related to the gap, and ideally populate effective ideas to address the gap. “By examining
what results you are achieving today and what results you want to accomplish tomorrow, you are
conducting a needs assessment” (Watkins et al., 2012, p. 19).
In the case of the LETC, the needs assessment’s purpose was to provide the CEO with
recommendations to improve the quality of their internal self-assessment used during the accreditation
process.
Goals
In order to achieve the needs assessment’s purpose, goals include the collection and analysis of
data to create a comprehensive set of performance solutions using human performance technology (HPT).
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
6
Directly tied to the needs assessment graduate students’ program, the HPT process was both
systematic and systemic. The needs assessment adopted a systematic methodical approach, as well as a
systemic consideration of how the focus impacted the whole organization. By driving a systematic
process, each step informed the questions and answers at the next level (Rossett, 2009, p. 42). Likewise,
although the focus of the needs assessment was at the operational level, inputs and outputs systemically
impact tactical and strategic levels, in addition to the interconnected systems at work in any organization,
including social systems.
Ethical considerations and professional standards were actively discussed and utilized throughout
the process. Particularly of concern was the political climate that may be roused with the needs
assessment and, the ethical factors of an internal team member with deeper knowledge and relationships
beyond this needs assessment.
Other goals included alignment of the recommended solutions at all levels of the organization,
even though the focus spotlighted the operational level. Most importantly, a goal was to provide solutions
that impacted results of the organization. The thought process of the team maintained focus on the
internal self-assessment with consideration to the ripple effect among the organization’s staff and
operations.
As a team, the goals included respectful practice and dialogue among internal team members and
with the client organization members. This was accomplished by discussing expectations at the beginning
of the project. Finally, the team vowed to learn about evidence-based needs assessment work, from the
collection and analysis of data to the solutions ideated by the team. Although professional “hunches” and
suspected issues were discussed, the team committed to only utilizing findings that were supported by
data-driven evidence.
Objectives
Based on the purpose and goals of the needs assessment, four objectives were established for the
needs assessment.
1. Given the available data, define the performance gap related to the internal self-assessment results
in preparation for federal accreditation.
2. Using systematic tools, identify causes of the performance gap through analysis of evidence
based data.
3. Once the causes have been identified, develop interventions that address the identified causal
factors.
4. Using the determined interventions and performance tools, prioritize the interventions to ensure
effective and efficient recommendations are made.
Define Needs
Methods Used to Identify Needs
Several HPT frameworks were used during the performance analysis including Joseph Harless’
front-end analysis model (1973), the Robert Mager and Peter Pipe performance analysis flow diagram
(1997, p. 5), Geary Rummler and Alan Brache’s three levels of performance (1995, p. 19), Watkins et
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
7
al.’s logic model (2012, p. 61), and Thomas Gilbert’s Behavior Engineering Model (1996, p. 83). Each
helped to evaluate the current and desired performance. All of these frameworks together provide a path
of analysis from the more granular details of the work, how those details connect to the operational goals
within the organization, and finally align to the more external impacts of the organization in its
community
Performance Analysis Flow Diagram
At the initial onset of the project, the performance analysis flow diagram (Appendix B, Figure 1)
was used as an efficient and effective way to examine what the performance gaps of the problem are and
whether it’s worth solving. In the case of this performance problem it appeared that there was a lack of
clear expectations, feedback was lacking, and consequences were misaligned. It also appeared that
resources were adequate and there wasn’t a genuine skill deficiency as evidenced in the success of
previous assessments. The data used to make these determinations came from observations, informal
interviews and a comparison of past and present assessment reports. This tool quickly revealed that the
performance problem was worthy to pursue. In a systematic way, the team could proceed through each
part of the performance analysis flow diagram and answer each question.
13 Smart Questions
The 13 Smart Questions (Appendix B, Figure 2) verified the information studied in the
performance analysis flow diagram. In addition, Harless’ questions helped to identify general types of
solutions, costs and development times, constraints, and overall goals. These questions encapsulate what a
manager, trainer, or performance technology consultant should ask before choosing solutions for a
performance problem (Harless, 1973, p. 231). The combination of the Performance Analysis Flow
Diagram and Harless’ 13 Smart Questions provided the team the evidence needed to continue pursuing
the performance problem.
Anatomy of Performance
The team used Rummler and Brache’s Anatomy of Performance (AoP) model (Appendix B,
Figure 3) to evaluate the LETC’s internal accreditation processes because this model can be used for
analyzing complicated issues that span multiple levels of an organization. The AoP illuminated that the
LETC is experiencing a process related performance deficiency, among other smaller issues, at the
operational level of the organization that has repercussions that span both the tactical and strategic levels
of the organization.
Logic Model
Logic models are common planning, monitoring, evaluating, and communicating tools used to
guide development projects (Watkins et al., 2012, p. 61). The logic model visually represents the
relationships among the resources operating in the LETC, the activities to plan, and the results the team
hopes to achieve. After completing the logic model, the team identified that the gap was located at the
operational level of the organization. Next, the team processed how the gap is connected to the tactical
and strategic levels of the organization. The logic model kept focus on the outputs and operational level,
while not straying out of scope when using data and interventions. By examining these relationships in
the project, the needs assessment team could determine the sequential connections that will deliver short
and long-term desired results.
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
8
Behavior Engineering Model (BEM)
The BEM was used to classify the root causes of the performance gap. Combined with other
models and frameworks, the BEM made sense to the project team, because the performance problem
primarily resides at the operational level of the organization. The BEM is a powerful tool to gather data at
the worker (operational) level and to distinguish whether performance problems are related to the person
or the environment. Additionally, the BEM dovetails easily into the nine performance variables of the
Anatomy of Performance (AoP) framework that allows for a clear connection to all three levels of the
organization, namely the worker, the process, and the organizational levels.
Data Identifying Needs
Using the methods to identify the needs the data was collected and scrutinized in a systematic
process to determine causes. The desired performance is linked to the operational, tactical, and strategic
goals of the LETC, which is also a goal of ABC Headquarters. The accreditation manager at the LETC
performs the work and is responsible to establish effective and efficient processes to conduct the work
(Accreditation Manager job description).
Current state of performance: 60% to 75% of the training accreditation standards are NOT met
during the internal self-assessment.
Desired state of performance: 100% of the training accreditation standards are met during the
internal self-assessment.
Evidence of the current performance is documented in two separate emails from Lee Clayton
(2013, September 25) and Ken Crooker (2013, September 24). Evidence was also collected in a face-toface interview with a third LETC assessor. Each assessor stated that only 25% to 40% of the standards
were met during the last internal self-assessment held in August 2013 (Clayton and Crooker, 2013
September). Personal observation of the accreditation files by Alan Wheaton substantiates that a low
percentage of files meet the standards. In addition, some assessors reported an overall decrease in the
quality of the BOSS files as compared to previous years. A comparison of past and present internal selfassessment reports indicates a decrease of compliant files per the ABC internal self-assessments (2013)
and the LETC Facilitator Course self-assessment report (2010).
The LETC is required to achieve federal accreditation every year. They have successfully
achieved this goal each year; however, it appears that some of the internal processes leading to
achievement of this goal are beginning to break down. Most notably is what appears to be a breakdown in
the preparation and feedback processes. During the internal self-assessments prior to the actual
accreditation, the LETC has internal experts conduct self-assessments of each account that needs to be
accredited. Over time the scores achieved during this process have declined and the assessors have
noticed that the same mistakes are being repeated. As a result of the poor results during the internal
assessments, the LETC is forced to expend massive amounts of resources in the form of staffing, money,
and time to ensure that the accounts are ready prior to the actual accreditation. This reactionary posture
has placed a strain on the organization’s resources. Further complicating this situation is the fact that the
LETC, even with the failed internal assessments, has still managed to achieve accreditation each year,
which provides a false perception that the accreditation process is working properly.
The desired performance of 100% of the training accreditation standards met would ensure the
files were ready for the assessors review and provide a more efficient, consistent process, saving the
assessors’ time during the internal self-assessment, and the accreditation managers’ time during the
correction of mistakes prior the actual self-assessment. Once the preparatory processes are improved, the
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
9
LETC can begin using the internal assessment process as intended, which is as a measurement tool that
determines the readiness of the LETC for accreditation, and not as method for fixing inefficient processes.
Based on the results of the logic model the team determined that the performance gap should be
addressed due to the importance of its impacts at the tactical and strategic levels.
Input
Activities
Outputs
(Operational)
Outcomes
(Tactical)
Impacts
(Strategic)
-time of the staff to conduct the internal self-assessment
-financial costs of staff and actual resources (technology, paper, facilities)
-procedures
-data collection tools
-databases
-external resources
-observations
-interviews
-evaluations
-benchmarking
-change management
-corrective action program
-peer reviews
-building the files with evidence for review
-collecting evidence for the files
-reviewing of the files by assessors
-the LETC meets 100% of the training accreditation standards during the internal selfassessment
-the LETC meets the BOSS accreditation standards
-supply the ABC with knowledgeable and skilled law enforcement professionals ready to
meet the demands of the nation at any time
-provide the persistent presence of law enforcement officers
-provide safety, security and environmental stewardship
-provide the community with a sense of protection and security
The team utilized multiple data sources and tools in order to acquire various views on the
performance gap (Watkins et al, 2012, p. 48). This included extensive document review, task analysis and
several interviews with key stakeholders. Following the data collection, the data was coded and further
analyzed using a task analysis and completion of an Ishikawa Fishbone Diagram to determine the root
causes of the flawed internal accreditation process. Data were triangulated with more than one source
when possible (Appendix D, Triangulation Data). The steps below show steps we took to analyze the
data.
First, the team started with interviews because there is no current task analysis for the internal
self- assessment. Interview questions were developed to obtain data about current processes and
performance gaps. The questions used to obtain this data are listed in Appendix E.
The data obtained in the interviews can be found in Appendices B and D. A brief summary of the
major points found during the interviews are listed below:
 There is no current consistent process for internal self assessments.
 Feedback is not provided.
 There is no incentive to change the current process.
Using the data from the interviews we completed a task analysis for the internal self-assessment process.
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
10
Using the data collected (interviews, observations, and document review) the team further
analyzed the problem by answering Rummler and Brache's questions as provided in the AoP model
(Appendix B, Figure 3). From that analysis, the team noted that several factors influencing the internal
accreditation process involved the Human Performance System.
To further investigate the causes of human performance deficiencies, an analysis was conducted
using the Factors Affecting Human Performance System (Wallace, 2013). This analysis was conducted
from the perspectives of the accreditation manager, the assistant accreditation manager, the internal
assessors, and the accreditation file managers(Appendix B, Figure 4).
The Ishikawa fishbone diagram then allowed the team to visually identify the root causes of the
problem. The problem- the flawed internal accreditation process- was placed to the right at the head of
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
11
the fish (per the model below). Just as the team used the BEM for data coding purposes, the team further
used the BEM’s six categories of Information, Instrumentation, Motivation, Knowledge, Capacity, and
Motives to organize the fishbone diagram. The secondary causal factors of poor performance were then
indicated on the smaller ribs parallel to the spine of the diagram. This simple depiction provides a clear
visual of all of the potential causes leading to the problem of the flawed internal accreditation process.
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
12
From the diagram the team could immediately see the causes, or what Rossett (2009) terms
“performance drivers...the factors that block or aid performance” (p. 49), such as available data, processes
and tools in the internal self-assessment, as well as motives and incentives. For example, no established
process is followed for the preparation and maintenance of the accreditation folders, nor is there a process
to communicate with all stakeholders about the status of each accreditation folder. The accreditation
manager, assistant accreditation manager, and assessors do not prescribe to any checklist that would allow
for consistent review of the folders. Regarding incentives, the assistant accreditation manager receives
unintentional positive incentives to perform poorly by others correcting the training standards for him
during the internal self-assessment. These causes are further described below in the identification of
interventions.
The relationship of the performance gap, the root causes, and the classification of the root causes
was also helpful to examine in the six cells of the BEM.
Performance Gap
~60% of the training
accreditation standards
are NOT met during the
internal selfassessment.
Data sources:
(Performers,
Supervisors, Policy
Documents, Other
Documents)
Data collection
methods: (Interview,
Observation, Document
Review)
Root Causes
No process for folder
management
No documented
internal selfassessment process
Key stakeholder job
tasks are unclear
No ongoing feedback
process
No tool used to track
and communicate
accreditation status
No specific checklist
used by assessors
Accreditation managers
awarded for poor
performance
No consequences for
poor performance
Accreditation managers
do not know how to
develop tracking tools
Assistant Accreditation
Manager is
overwhelmed with the
accreditation processes
Master Training
Specialists are no
longer available to
provide additional
support to the Assistant
Accreditation Manager
Internal self- assessors
are no longer
volunteering
Classification (BEM)
Data
Feedback
Environment
Support
Tools
Resources
Incentives
Knowledge
Skills
Capacity
Person
Motives
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
13
Interventions
Methods for Identifying Alternative Sets of Interventions
After identifying the needs, defining the performance gap, and evaluating the causes of the
performance gap, interventions were explored. To identify interventions, the team processed verbally at a
synchronous meeting rather than through electronic, asynchronous communication to allow brainstorming
and quick dialogue of possible solutions. This allowed accurate understanding of each idea and
perspective, as well as synergy through the dialogue as we reviewed the intersections and patterns of the
data.
First, the team reviewed methods to identify alternative sets of interventions. The BEM and the
Anatomy of Performance (AoP) models provided structure to do this. For example, many data points
appeared in the environmental supports section of data. Problems with lack of process and
communication jumped off the BEM as primary issues. Considering the related organizational elements
from the AoP, solutions for these problems would be supported by senior leadership and be low cost to
the LETC. At the organizational level, any solutions must efficiently utilize resources and make sense.
In addition to the BEM and AoP, Appendix A4 in Watkins, West Meiers and Visser (2012) was
utilized throughout the needs assessment. Particularly at this stage, the “select potential solutions” section
for organizational needs and team performance encouraged the use of the multicriteria analysis (p. 271272), which is noted below in the criteria used to compare interventions.
Data and Criteria Used to Compare Interventions
Given the fact that the team identified 12 separate interventions, the multicriteria analysis was
used to determine which ones should be used to close the performance gap. Watkins et al. (2012) note
that the multicriteria analysis provides a systematic method to quantitatively compare multiple
interventions (p.171).
As part of the multicriteria analysis the team had to identify the appropriate criteria to consider
when scoring each intervention. Watkins et al. recommend considering five to eight attributes for this
process (2012, p. 173); the team decided upon five. The attributes the team considered were: expected
results after six months, feasibility of implementation, cost of activity over first year, ability to
accomplish desired outcomes, and acceptability.
Deciding upon appropriate attributes to consider was critical to distinguish among the multitudes
of interventions. ‘Expected results after six months’ was chosen as an attribute because it would provide
an idea of which interventions would have immediate and substantial impacts. The ‘feasibility of
implementation’ provided a reality check as to whether a considered intervention was reasonable to
implement. The ‘cost of activity over the first year’, which is also linked to feasibility, provided an idea if
the intervention could be implemented at a low cost, or if it was cost prohibitive.
Perhaps the most important attribute considered, was the ‘ability to accomplish desired
outcomes,’ which took into account the ability of the proposed intervention to close the performance gap.
Lastly, ‘acceptability’ was considered from the perspective of the organization’s likelihood of embracing
the proposed intervention. Considering that the LETC is a vertically structured organization that closely
follows the chain of command, ‘acceptability’ was weighted based on the perceived likelihood of the
LETC’s leadership accepting the intervention.
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
14
After identifying the appropriate criteria to distinguish between the interventions, the team
considered each intervention individually and scored each attribute between one and ten (one being the
least effective and ten being the most effective). After scoring the five attributes for each individual
intervention, the scores were totaled and divided by five. This provided an average score for each
intervention and provided a simple method to compare the interventions.
Interventions and criteria scores (note a more thorough explanation of each intervention is
provided later in this report):
Interventions Criteria for Multicriteria Analysis
Expected
results
after six
months
Feasibility of
Implementation
Cost of
activity
over
first year
Ability to
accomplish
desired
outcomes
Acceptability
Accreditation Job aid/
process flow
5
6
7
10
Updates to LETC
Organization &
Regulations Manual
2
7
6
10
1
0
7.0
Dashboard for
tracking accreditation
status
7
9
9
9
1
0
8.8
Checklists (for
accreditation
preparation
processes)
8
8
9
9
1
0
8.8
Performance
Management
(expectations/
accountability)
5
5
9
7
8
6.8
Schedule for
assessors
3
3
7
6.6
Incentives for
assessors
3
1
0
05
8
4
5
5
Dispute resolution
process
5
7
9
7
7
7
Documented role
clarity with task list
2
9
5
9
7
Onboarding
checklist for
stakeholders
3
5
6
7
8
5.8
SharePoint training
(For AM/AAM/
Stakeholders)
5
9
8
8
9
7.8
Community of
Practice
1
3
8
6
5.6
1
0
1
0
1
0
9
Average
Score
7.4
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
15
Prioritized List of Interventions
Based on the above criteria comparing possible solutions, the interventions were discussed and
prioritized. Each intervention possesses advantages and disadvantages.
1. Process and Communication Flowchart: Multicriteria Score 7.4 (Classification- Environment/
Data)
A process and communication flowchart, or an algorithmic visual display, of the internal selfassessment will provide all stakeholders an important reference. The flowchart must include all steps as
well as the communication and feedback points from the point of preparing folders, to assessing the
folders, to what happens after the actual self-assessment. To assist accountability and true buy-in of the
process, the flowchart should be vetted with senior leadership at the LETC, and then be displayed for all
in an agreed upon shared site, either physically in the office or/and electronically via SharePoint, as well
as maintained in the organization’s manual. Visio could be a program to use that would allow the flow
chart to reflect the sequencing and if/then scenarios during the process.
Ideally, this intervention would envelope other interventions noted, such as the checklist created,
established schedule for assessors, documented role clarity, and a dispute resolution process.
Data from document review, multiple interviews and the task analysis concur that no consistent
process or exchange of communication exists (Appendix D, Triangulation Data). Currently, the AAM
collects the evidence when an assessment is nearing, and the assessors conduct the internal selfassessment from their prior knowledge of the training standards. The assessors receive no feedback to
whether what they provided was needed (or appreciated) and are confused when the same mistakes occur
in folders multiple times. Others involved in the process share information on accreditation status verbally
in update meetings, however, most are not aware of the status or upcoming needs for assessments. The
flowchart provides the roadmap to solve this significant performance gap.
2. Updates to LETC Organization and Regulations Manual (OrgMan) and Documented Role
Clarity: Multicriteria Score 7 (Classification- Environment/ Data)
Recommend that the organization include BOSS standards numbers when there is an update to
the LETC OrgMan. By ensuring BOSS is included in the update means that all changes will comply with
both accreditation and the organization standards. This will prevent loss of accreditation standards while
maintaining organizational standards. If a committee is included in the process of updating the OrgMan,
the loss of accreditation standards will be prevented. The committee will include all contributing key
stakeholders to the accreditation process, and they will meet to approve any changes. Before approving
any change the key stakeholders will be able to review and edit proposed changes to the OrgMan.
Role clarity with a task list for all the key stakeholders is also recommended for this intervention.
By having a written task list for each stakeholder, the process will not become clouded or overwhelm any
one person. Each stakeholder will be aware of his/her specific contribution to the update process in the
OrgMan. This will prevent confusion about what needs to be done to update the OrgMan as well as who
participates in the process as the turnover rate is high at the LETC. There will be no confusion left about
what specific role they are expected to complete and what each role entails. This will benefit both the
proposed changes and stakeholders involved.
3. Dashboard for Tracking Accreditation Process: Multicriteria Score 8.8 (Classification Environment/Data)
Recommend that the organization use a dashboard or other dynamic display which will provide
all accreditation stakeholders with awareness of the current status of all critical accreditation milestones.
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
16
As mentioned by several stakeholders during the interview process, many of the personnel involved in the
accreditation process never know what the status of the greater process is. Instead, they are typically
notified at various times during the process and asked to provide assistance in a reactionary capacity.
By using a display in an area where all stakeholders, including leadership, can see a current
snapshot of the current accreditation status, all stakeholders will then be empowered to take action prior
to milestones becoming delinquent. The dashboard would be displayed on a large monitor or whiteboard
in the main briefing room for LETC personnel to see on a daily or weekly basis.
4. Checklists: Multicriteria Score 8.8 (Classification - Environment/Tools, Resources)
Recommend the organization coordinates with other accredited agencies to develop a standard
checklist to be used throughout the entire accreditation community. If this is not feasible, the organization
should create their own checklist to prepare for and conduct internal self-assessments.
From 2003-2013 standardized checklists were used by accreditation managers as a job aid to
populate folders and were also used by assessors to evaluate each folder/standard. These checklists were
provided by BOSS and were required to be submitted to BOSS with the final assessment report. In the
summer of 2013 BOSS removed the requirement to fill out and submit checklists for each folder. In fact,
BOSS removed the checklist from their website and the checklists are no longer available. BOSS
removed this requirement because they were unable to administratively handle the amount of checklists
received. As a result, some agencies have created their own checklists for internal use. These checklists
are not standard amongst each agency. Over the past 10 years the use of a standardized checklist has
proven to be an effective tool (job aid) for the accreditation manager to prepare folders and the
assessment teams to objectively evaluate each standard. The checklists were also used to by the assessors
to provide objective feedback to the accreditation manager based on the standardized criteria.
5. Performance Management: Multicriteria Score 6.8 (Classification – Environment/Incentives)
Recommend that the organization incorporate completion of accreditation tasks as part of each
stakeholder’s greater performance goals and evaluations. From interviews with several stakeholders, it
was determined that positive and negative feedback relating to individuals’ participation in the
accreditation process is not factored into performance goals and appraisals.
This process would involve all supervisors communicating their expectations for the accreditation
process to each subordinate involved with accreditation. This would further involve each subordinate
communicating to their supervisors what tasks they have accomplished in completing their part of the
accreditation process. By doing this, ownership of the entire process is expanded from a few individuals
to all stakeholders involved. This further ensures expanded accountability which will lead to better care
taken to ensure accreditation folders are maintained in accordance with BOSS standards.
6. Set Schedule for Assessors: Multicriteria Score 6.6 (Classification- Environment/Incentives)
Recommend that a set schedule be made for assessors to participate in the self-assessment.
Currently assessments are done by volunteers and the volunteers (low number) are repeatedly the same.
Having a set schedule will ensure everyone contributes equally to the self-assessment preparation as well
as creating diversity among assessors. By rotating assessors, the same person is not repeatedly sacrificing
their allotted work hours to review folders, and the same person is not repeatedly contributing their
knowledge. The schedule will include all personnel and rotate for each assessment preparation.
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
17
7. Incentives for Assessors: Multicriteria Score 5.0 (Classification – Environment/Incentives)
Recommend assessors that volunteer for internal self-assessments are incentivized. The
organization has mechanisms readily available such as; time off awards, letters of appreciation, public
recognition, cash awards, and performance appraisals, to name a few.
Recently, several internal assessors have quit volunteering their time to conduct internal selfassessments. These assessors are the same ones that normally volunteer and have done so for several
years. These assessors not only conduct internal assessments but also conduct assessments for other
agencies; they are experts at this job. The internal assessors stopped volunteering because they don’t
receive any feedback and they feel their time and expertise is not valued. Since the assessors quit
volunteering, a recent internal self-assessment was cancelled. As a result of not conducting an internal
self-assessment, the external self-assessment team (the next step in the process) reported numerous
deficiencies with many of the standards.
8. Dispute Resolution Process: Multicriteria Score 7.0 (Classification – Environment/Incentives)
Recommend that an informal dispute resolution process be created to provide a method for
ensuring divergences of opinions between the assessors and the assistant accreditation manager (AAM)
be identified and resolved quickly during the internal accreditation process.
During the interview process it was discovered that the assessors had become frustrated during
the internal assessment process when they noticed that the same discrepancies that they had already
identified during earlier assessments reappeared during subsequent internal assessments. They also noted
that the assistant accreditation manager did not communicate to them which specific recommendations
that they had provided that he was going to incorporate to ensure the folders were ready for accreditation.
This was confirmed in an interview with the AAM when he verified that he does not provide the assessors
with feedback and at times he chooses not to implement their feedback.
This intervention can only be implemented after intervention 1 (above), process and
communication flowchart, is implemented. This means that the assessors must be informed following
each internal self-assessment regarding what feedback is being implemented or not implemented after
they conduct their internal assessments. They will then consult with the assistant accreditation manager
on items that they did not agree upon and then come to a final mutual agreement on how similar situations
will be handled in the future. In the rare event that the AAM and the assessors cannot come to an
agreement, the final resolution can be determined by the AM or the training officer. Upon resolution, the
results need to be communicated to the appropriate stakeholders to ensure that all LETC parties have a
shared understanding of the accreditation process moving forward.
9. Documented Role Clarity: Combined with #2
10. Onboarding: Multicriteria Score 5.8 (Classification- Person/ Knowledge)
Establishing knowledge of and expectations for the accreditation process right when staff begin
employment at the LETC is the goal with adding this item to any existing onboarding checklists utilized
at the organization. Just as the organization would cover topics related to job expectations, benefits, and
organizational standards, onboarding can include discussion of the accreditation process and reference to
the process and communication flowchart that includes expectations for all staff involved.
At this time, not all involved in training are aware of the training standards or accreditation
process at all. Even at the leadership level, there exists a frustration with how much time is spent on the
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
18
accreditation process, as verified in interviews, resulting in the work happening at times “under the
radar.” Without true leadership support and understanding of the process, those involved have little
understanding or misinformation. Onboarding would create a clear picture from the start of the
requirements.
11. SharePoint Training: Multicriteria Score 7.8 (Classification – Person/Knowledge)
Recommend providing SharePoint Training for the assistant accreditation manager, the
accreditation manager, and other accreditation stakeholders. During the data collection process it was
noted that there is not a standard project management program for managing the accreditation process.
SharePoint has been identified as a viable resource for overseeing the accreditation process and it is
currently in use by the BOSS organization which is the largest federal law enforcement training center in
the U.S. and is also being used by several of LETC’s partner organizations. Further, SharePoint is
currently being used by several of LETC’s stakeholders for other non-accreditation purposes. However,
the personnel identified as key managers for the accreditation process currently do not have the skill base
necessary to leverage SharePoint as a process management tool.
The team recommends that the LETC work with a professional SharePoint trainer to ensure that
the managers learn how to use SharePoint to manage the accreditation process. In order for this training to
be successful and also lead to continued use, the team recommends that the training for the accreditation
manager and assistant accreditation manager go beyond simple use of the program, and actually
incorporate how to create a functional accreditation management project interface with all stakeholders
involved; they can then allow the managers to use the tool as they will in the future. As Peggy Ertmer and
Timothy Newby state in their article Behaviorism, Cognitivism, Constructivism: Comparing Critical
Features from an Instructional Design Perspective, “the ultimate measure of learning is based on how
effective the learner's knowledge structure is in facilitating thinking and performing in the system in
which those tools are used” (1993, p. 64). This training will be most successful in achieving transfer in
the workspace if the managers learn to use the program as they will actually use it at the LETC. This
intervention will also require the support of LETC leadership in setting the appropriate expectations that
SharePoint be used as the primary accreditation management tool in the future.
12. Community of Practice: Multicriteria Score 5.6 (Classification- Person/ Motives)
The needs assessment team discussed this as the most controversial potential intervention, as the
culture of the LETC has not fully embraced accreditation so establishing a community of practice may
require a shift in the culture. A shared exchange of the challenges and opportunities of the internal selfassessment and accreditation process overall could be a motive for involved staff by sharing the load,
increasing collegial interactions and ideating future improvements together.
At this time, the relationship between the accreditation managers and the assessors has become
somewhat tenuous. Misunderstandings exist on both sides. A community of practice would provide an
informal learning opportunity for all involved to access one another. Wenger (2009) describes that a
community of practice reveals “emergent structure, complex relationships, self-organisation, dynamic
boundaries, ongoing negotiation of identity and cultural meaning” (p. 179-180). The expectation would
not be that the community of practice will immediately tie the LETC staff together, but it would allow an
informal framework to allow the relationships to grow with common goals of improving the internal selfassessment and the overall accreditation process.
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
19
Recommendations
The needs assessment produced several recommendations from the team, which address the
operational, tactical and strategic goals of the organization. Due to the causes of the performance
deficiency, the team identified and recommends several performance interventions that would remove the
barriers to desired performance at the LETC.






Checklists for accreditation preparation processes (Classification Environment/Tools, Resources)
Data for tracking accreditation processes (Classification – Environment/Data)
SharePoint Training (Classification – Person/Knowledge)
Process and Communication Flow Chart (Classification – Environment/ Data)
Updates to LETC Organization and Regulations Manual (Classification – Environment/ Data,
Instruments)
Dispute Resolution Process (Classification – Environment/ Data, Instruments)
At the operational level, these recommendations plug the holes of lack of process, communication,
and tracking. All staff directly involved in the internal self-assessment will benefit by following agreed
upon processes, communication loops, and maintaining updated status reports via a tracking tool. In
addition, a dispute resolution process, even an informal one, establishes an agreed-upon procedure when
there may be confusion or disagreement throughout the process.
Likewise at the tactical level, by ensuring these processes establish consistency and preparedness, the
internal self-assessment is more likely to successfully renew accreditation at the agency level. The LETC
strives to ensure their courses prepare staff to the highest performance. Setting up the accreditation
process for success accomplishes this tactical level goal.
The intervention recommendations address the strategic level of the organization, as well. The
operational and tactical goals align with the strategic goal of maintaining an agency that instills “a high
degree of public confidence in the competence and professionalism of federal agents and officers” (BOSS
procedures and standards manual, 2010, p. 3). A small example is the tracking tools recommended, which
would allow transparency of the organization’s status, in addition to keeping all levels of the organization
informed.
The recommended interventions directly impact all three levels of the organization, which the team
focused on from the beginning of the needs assessment. The recommendations would produce efficient,
consistent process and communication among staff about the internal self-assessment, while setting up the
accreditation reviews successfully, which in turn meets the organization’s commitment to the community.
Last, the needs assessment team recommends the organization develop a comprehensive plan to
implement the alternative sets of interventions and continuously monitor and evaluate the performance of
the internal self-assessment. While these six interventions were chosen based on their potential for a
significant impact on closing the performance gap, if resources are available, the LETC should also
consider implementing the remaining interventions from the multicriteria analysis.
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
20
References
Clayton, L. (2013, September 25). Re: Needs assessment question (BOSS selfassessments) [Electronic mailing list message]. Retrieved from ABC.com email server.
Crooker, K. (2013, September 24). Re: Needs assessment question (BOSS selfassessments) [Electronic mailing list message]. Retrieved from ABC.com email server.
Ertmer, P. A., & Newby, T. J. (1993). Behaviorism, cognitivism, constructivism: Comparing critical
features from a design perspective. Performance Improvement Quarterly, 6(4), 50-72. doi:
10.1111/j.1937-8327.1993.tb00605.x
Gilbert, T. F. (1996). Human competence: Engineering worthy performance (Tribute ed.). Silver Spring,
MD: International Society for Performance Improvement. (Original work published 1978).
Governing Body of the LETC Accreditation (2010). BOSS Standards and
Procedures Manual. Retrieved from http://www.ABC.com
Harless, J. (1973). An analysis of front-end analysis. Improving Human Performance:
A Research Quarterly, 4, 229-244.
LETC (2010, April 13-14). Self-assessment Report for the The ABC
Law Enforcement Training Corporation Facilitator
Course Accreditation. Retrieved from the LETC database.
Mager, R., & Pipe, P. (1997). Performance analysis flow diagram. In Analyzing
performance problems, or you really oughta wanna: How to figure out why
people aren’t doing what they should be, and what to do about it (third edition) (p. 5). Atlanta,
GA: Center for Effective Performance.
Position Description (2010, February 4). Organization title: assistant training officer
and accreditation manager. Retrieved from LETC database.
Rossett, A. (2009). First things fast: A handbook for performance analysis (second edition). San
Francisco: Jossey-Bass/Pfeiffer.
Rummler, G. A., & Brache, A. P. (2013). Improving performance: How to manage the
white space on the organization chart (3rd ed.). San Francisco, CA: JosseyBass.
Schensul, J., & LeCompte, M. (2013). Essential ethnographic methods: A mixed
methods approach (2nd edition). Lanham, MD: AltaMira Press.
Wallace, G. (2013, February 3). Geary A. Rummler on performance engineering 1986.
Retrieved from http://www.youtube.com/watch?v=XrzmZLislDs
Watkins, R., West Meiers, M., & Visser, Y. (2012). A Guide to Assessing Needs:
Essential Tools for Collecting Information, Making Decisions and Achieving
Development Results. Washington, D.C.: The World Bank.
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
21
Wenger, E. (2010). Communities of practice and social learning systems: The career of a concept. In C.
Blackmore (Ed.), Social learning systems and communities of practice (pp. 179-198). London:
Springer.
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
22
Appendix A: Supporting Data Inventory List
(Note: The supporting documents are located in a separate PDF file)
Item
Description
PDF Page Numbers
Note 1: Yellow highlighting was used to indicate relevant data. Specific page numbers are indicated by an asterisk - 40 pages
contain highlighting.
Note 2: Text boxes (also highlighted in yellow) were used to provide additional comments by the internal analyst.
1
Interview: TO
*1
2
Interview: Assistant Accreditation Manager
*2 - *3
3
Interview: Accreditation manager and an Internal Assessor
*4, *5, *6, & *7
4
Interview: Internal Assessor
*8 - *9
5
Intentionally left blank
Intentionally left blank
6
Interview: External Team Leader Assessor
*10-11
7
Email: Internal assessor
*12
8
Email: Internal assessor
*13
9
ORGMAN Chapter 1
(Unit Organization)
15-28 (*16, *21, *23,
*24)
10
ORGMAN Chapter 4.
(Training Division)
29-40 (*32, *33, *39, *
40)
11
ORGMAN Chapter 5.
(Performance Systems Division)
*41
12
Standards Manual
42-47 (*45, *46, *47)
13
Training material for Accreditation Managers
48-52 (*49)
14
Accreditation Schedule
*53
15
Accreditation Tracking Document
55-68 (*55)
16
Position Description (Accreditation Manager, GS-13)
*69
17
Position Description (Assistant Accreditation Manager, Contractor)
*70
18
Standards Responsibilities Chart
*71
19
AM Accreditation Checklist
72-76 (*72, *74)
20
Accreditation Status Report
77-81 (*77)
21
Assignment to Duties Instruction
82-89 (*82)
22
Intentionally left blank
Intentionally left blank
23
2006 Program A Self Assessment Report
90-100 (*90, *93)
24
2010 Program B Self Assessment Report
101-111 (*101, *103)
25
August 2013 Academy Self Assessment Report
112-138 (*113)
26
November 2013 Program C Self Assessment Report
139-141 (*139)
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
23
Appendix B: Performance Technology Frameworks
Figure 1: Mager and Pipe - Performance Analysis Flow Diagram
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
24
Figure 2: Harless - 13 Smart Questions
1. Do we have a problem?
Yes.
2. Do we have a
performance problem?
Yes.
3. How will we know when When accreditation is achieved using less staff hours and less interruptions
the problem is solved?
to workers regular work schedule. Other projects will be completed on time
and quality of other projects will improve.
4. What is the performance Work is being conducted inefficiently causing other projects to missing
problem?
deadlines and/or with reduced quality.
5. Should we allocate
resources to solve it?
Yes.
6. What are the possible
causes of the problem?
Expectations, roles, feedback, are unclear. Processes and procedures are
ineffective.
7. What evidence bears on
each possibility?
Workers are complaining that their feedback is not used to mitigate
recurring problems. Other projects are being rushed or timelines are moved.
Quality of work is reduced.
8. What is the probable
cause?
Processes and procedures are unclear or nonexistent.
9. What general solution
type is indicated?
Create a process chart and assign various parts to the appropriate
stakeholders.
10. What are the alternate
subclasses of solution?
Use project management techniques to manage each accreditation project.
Regularly communicate with key stakeholders. Conduct debriefs to identify
lessons learned and best practices. Consider using SharePoint as the
document management system.
11. What are the costs,
effects, and development
times of each solution?
Costs include staff hours to create new processes, communicate processes,
and develop the skills to manage SharePoint.
Effects: The accreditation manager will be required to work with a
performance technologist to create process charts, etc. and learn how to use
SharePoint (if used).
Six months to implement new processes and learn how to use SharePoint.
12. What are the
constraints?
Willingness of the accreditation managers to accept and learn new processes
and procedures.
13. What are the overall
goals?
Develop alternative sets of interventions to maintain LETC accreditation.
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
25
Figure 3: Anatomy of Performance, Three Levels of Performance
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
26
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
27
Figure 4: Anatomy of Performance, Human Performance System
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
28
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
29
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
30
Appendix C: Lessons Learned
In hindsight, the needs assessment graduate school team members identified some things they
would have done differently during the project process. One of the lessons learned is to communicate and
develop partnerships with the CEO and all key stakeholders of the client organization throughout the
entire project. Even though this was a school project, the client and those closest to the work should feel
as if they own (or at least acknowledge) they have a performance problem. Otherwise the data collection
efforts may be seen as in invasion of their work place. It was also discovered that some of the people
closest to the work opine there is not a performance problem or an opportunity for improvement and that
they might be at risk as part of this assessment. Rossett (2009) warns of this and recommends to
“understand your sources” in order to prohibit misconceptions (p. 127). Developing a “cover story” at the
very beginning of the project would have been an effective tactic to ensuring the client was on board with
the goal of the needs assessment. The team established a cover story prior to data collection and
interviews, however, in hindsight one would have aided establishment of the relationship from the start.
Establishing a communication plan seems like an obvious thing to do, however, the project moved so
quickly that this was not a priority. Midway through the project and once the interviews were underway
with the managers, it appeared as if there was some friction between the partnership. Fostering a healthy
partnership with the client organization would increase the likelihood that the interventions would be
implemented with the ultimate goal of adding value to the organization.
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
31
Appendix D: Triangulation of Data
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
32
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
33
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.
34
Appendix E: Interview Questions
1. Al has explained that BOSS may be onsite or offsite to do the actual self assessment, but as far
as preparing for the internal self assessment, walk us through the process you and your team
follow to prepare for it?
2. Have you always prepared the folders the way you do today?
3. What do you do with the feedback from the assessors of the internal self assessment?
4. How do you track the status of the standards and folders?
5. How do Jones and other senior leaders support your efforts?
6. What would you change about the process to make it more effective?
7. What training were you provided or did you do to prepare for the accreditation process?
8. As I said, the LETC is seen as a leader in accreditation among all agencies. What are your
expectations for Durango and Jones in succeeding in this important process?
9. What is your opinion of how the process runs today?
10. How are you updated on the status of accreditation and reaccreditation of the courses?
11. What would be the consequence if for some reason the LETC did not pass accreditation or
reaccreditation of a course?
12. Do Durango and Jones have the capacity to do what is needed for this process?
13. How do you support the accreditation process?
© OPWL 2013. Do not distribute without written permission from the OPWL department at Boise State
University.