COGNITIVE ASSESSMENT POLICY AND PRACTICE
TRAINING FOR MANUFACTURED CONSTRUCTION (TRAMCON) CONSORTIUM
ROUND 4 TAACCCT GRANTEE
JONATHAN W. ELLIOTT, PH.D.
April 2016
This deliverable satisfies Core Element 2: Career Pathways, Strategy 1, Activity 3.1.
This workforce product was 100% funded by a grant awarded by the U.S. Department of Labor’s Employment and Training
Administration. The product was created by the grantee and does not necessarily reflect the official position of the U.S.
Department of Labor. The U.S. Department of Labor makes no guarantees, warranties, or assurances of any kind, express
or implied, with respect to such information, including any information on linked sites and including, but not limited to,
accuracy of the information or its completeness, timeliness, usefulness, adequacy, continued availability, or ownership.
This work is licensed under a Creative Commons Attribution 4.0 International License.
Knowledge, Skills, Abilities, and Other (KSAO)
Purpose and Definitions:
This report satisfies Deliverable 2, Section 3.1, of the TRAMCON proposal. The data collected from TRAMCON
participants are extensive, in that they cover broad student characteristics in a series of instruments that
measure or assess the student’s Knowledge, Skills, Abilities, as well as Other (KSAO) traits. In the case of
TRAMCON, these four broad areas are grouped under three general constructs as defined below:
Knowledge: TRAMCON classifies knowledge as the requisite understanding of the manufactured constructionspecific training content covered in each level of training (Foundation, Basic, Advanced and Supervisory).
Appropriate content knowledge for each level of training was determined through collaboration with key
industry members and national certifying organizations that specialize in manufacturing, construction and
TRAMCON’s target job-placement market of manufactured construction (MC). In laymen’s terms, these
categories of assessment include instruments and tests that determine the participant’s level of understanding
of the critical concepts, industry practices and terminology required to enter, and perform well in, the field of
manufactured construction.
Skills and Abilities: TRAMCON assesses student abilities and skills using hands-on performance demonstrations,
tests and assessments. Students are required to physically perform construction and manufacturing-related
tasks at given levels of proficiency in order to successfully complete the training and receive certifications. The
appropriate level of participant skill and ability is based on the criteria of existing national certifications. For
example, the National Center for Construction Education and Research (NCCER) has identified necessary
construction-related skills (such as the use of hand tools) and the proficiency levels required to achieve the
NCCER Carpentry Level 1 Certification. These performance tests are included in the majority of NCCER training
modules. Table 1 provides an example of NCCER Construction Core performance tests that are conducted as
part of the TRAMCON Foundation Level Training. Passing of the performance tests for national certifications
constitute a trainee’s demonstration of proficiency in critical skills and abilities required of TRAMCON
graduates.
Table 1: Foundation Level Example: Performance Tests
TRAMCON
Training
Level
Foundation
Foundation
NCCER
Construction
Core
1
NCCER
Module
ID
00101-15
NCCER Module Name
Basic Safety (Construction Site Safety Orientation)
NCCER
Derived
Performance
Test
Yes
2
00102-15
Introduction to Construction Math
No
Foundation
3
00103-15
Introduction to Hand Tools
Yes
Foundation
4
00104-15
Introduction to Power Tools
Yes
Foundation
5
00105-15
Introduction to Construction Drawings
Yes
Foundation
7
00107-15
Basic Communication Skills
Yes
Foundation
8
00108-15
Basic Employability Skills
No
Foundation
9
00109-15
Introduction to Materials Handling
Yes
Other: TRAMCON assesses other student characteristics shown in occupational and education research to
enhance student learning and decrease training program attrition. These psychological and learning related
characteristics include both stable and dynamic traits such as manufactured construction domain-level selfefficacy and motivation as well as resilience (e.g. grit) and learning style. It should be noted that participant
responses to psychological construct survey items have neither correct nor incorrect answers. Rather, these
data are used to determine/identify students who may be more likely to drop out of training. TRAMCON
instructors can use student learning style data to gain an understanding of an individual student’s (or group of
students in a training cohort) characteristics in order to better align content delivery with preferred learning
method and/or identify students who may need additional assistance. The constructs in the “Other” category
can help training organizations and colleges identify students who may benefit from additional support services
intended to increasing training program completion rates.
Data Sources, Linking Data Sources, and KSAO Constructs
Each category of the KSAO assessment includes multiple constructs and data collected from various sources.
These data are compiled by the TRAMCON consortium into a comprehensive database; the Education to
Outcomes (ETO) software interface. Since data are collected from multiple sources, and at different times
during the training, linking of the student data between assessments is critical. Data linking is accomplished
through the collection of three unique participant identifiers. These identifiers include the participant’s college
of attendance, date of birth (DOB), and a unique college-assigned student identification number. It should be
noted that the participant’s personal data (i.e. name, social security number, etc.) are not included in the KSAO
data collection instruments, rather the college data collection and/or recruitment specialists are responsible
for linking the KSAO data collected to the individual students who are attending their respective college and
participating in the TRAMCON program.
Each KSAO assessment includes multiple constructs, however not all constructs are included in each and every
KSAO assessments. Rather, the inclusive list of KSAO constructs below are assessed at strategic points in the
training curriculum. Further information regarding the measurement of KSAO data, including which constructs
are assessed at specific times in the program, is provided under the Instrumentation section of this report. The
constructs included within each category of the KSAO and their respective definition under the TRAMCON
program are as follows:
Knowledge Constructs:
NCCER Content Questions: The NCCER KSAO content questions have been extracted from the NCCER text books
and the presentation slides covered in the TRAMCON instruction. The content of NCCER questions are tailored
to the specific NCCER modules included in each level of the training. The NCCER content questions in the KSAO
are not used as the means for certifying students in NCCER Modules. That is, TRAMCON students are required
to pass NCCER-generated exams that are administered by a NCCER certified trainer or training center in order
to receive the NCCER certifications. NCCER items within the KSAO are used to determine student knowledge
of NCCER content at pre- and post-training intervals.
Manufacturing Skill Standards Council (MSSC) Certified Production Technician (CPT) Content Questions:
The MSSC CPT KSAO content questions have been extracted from the MSSC student handbooks and the
presentation slides covered in the TRAMCON instruction. The content of MSSC CPT questions are tailored to
the specific MSSC CPT modules included in each level of the training. In should be noted that the MSSC CPT
content questions in the KSAO are not used as the means for certifying students in MSSC CPT. That is,
TRAMCON students are required to pass MSSC-generated exams that are administered by MSSC certified
training centers in order to receive the MSSC CPT certification. MSSC CPT items included within the KSAO
surveys are used to determine students’ knowledge of MSSC CPT content at pre- and post-training intervals.
TRAMCON Manufactured Construction (MC) Level 1 and Level 2 Content Questions: The MC KSAO content
questions have been extracted from the MC student handbooks and the presentation slides covered in the
TRAMCON instruction. MC Level 1 and Level 2 items included within the KSAO surveys are used to determine
students’ knowledge of MC content at pre- and post-training intervals.
Skill and Ability Constructs:
NCCER Hands-on Proficiency Demonstrations/Performance Tests: TRAMCON students are required to pass
NCCER-generated performance tests that are administered by a NCCER certified trainer or training center in
order to receive the NCCER certifications. NCCER items within the KSAO are used to determine student
performance on NCCER content at pre- and post-training intervals.
Other Constructs:
Manufactured Construction Training Domain-Level Self-Efficacy: According to Bandura (1986), self-efficacy is
one’s judgments of their capabilities to organize and execute courses of action required to attain designated
types of performance. Self-efficacy was operationalized within the domain of construction training by Elliott
and Lopez del Puerto (2015). Manufactured construction training domain-level self-efficacy survey items can
be found in Appendix A.
Manufactured Construction Training Domain-Level Motivation: Motivation and motivated behavior,
respectively, are defined by Eagle (2011) as an individual’s reason(s) for carrying out an action or a behavior
intended to accomplish a particular purpose. Motivation was operationalized within the domain of
construction training by Elliott and Lopez del Puerto (2015). Manufactured construction training domain-level
motivation survey items can be found in Appendix B.
Manufactured Construction Training Domain-Level Planned Behavior: This construct, rooted in Ajzen’s (1991)
Theory of Planned Behavior, contained two of the theory’s constructs; intentions and perceived behavioral
control. Intentions are defined as the perceived ease or difficulty of performing a behavior that reflects past
experience as well as anticipated impediments and obstacles. Planned behavioral control constitutes a person’s
perception of the ease or difficulty of performing a behavior of interest. Planned Behavior was operationalized
within the domain of construction training by Elliott and Lopez del Puerto (2015). Manufactured construction
training domain-level planned behavior survey items can be found in Appendix C.
Grit/Resilience: Grit is defined as one’s perseverance and passion for long-term goals (Duckworth, Peterson,
Matthews and Kelly (2007). The grit scale (Appendix D) is a generalized measure and is not specifically
operationalized to the manufactured construction domain in the KSAO.
Learning Style Inventory: The Learning Style Inventory (LSI) comprises a block of questions that TRAMCON
participants complete. This measure (Appendix E) is intended to help respondents (and their instructors)
identify their learning style preferences (Tactical, Auditory, and Visual) as well as provide strategies that may
benefit a learner of a particular style in retaining the topical content covered in an educational environment.
The LSI (Odessa, N.D.) utilized in TRAMCON is a generalized assessment and is not specifically operationalized
to the manufactured construction training domain in the KSAO.
Data Collection/Survey Administration Process
KSAO data are collected at several intervals during the training. In general, constructs are measured at
participant intake, as well as pre- and post-training module completion. While the program contains various
participant entry and exit points, Figure 1 provide a flow chart describing the KSAO instruments administered
to a TRAMCON participant who progresses through the entire training curriculum.
Figure 1: TRAMCON Training and Associated KSAO Instrument Assessment
Instrumentation
As seen in Figure 1, there are nine KSAO instruments administered within the TRAMCON curriculum. As
previously noted, not all KSAO constructs are administered in every survey. Rather, selected constructs are
administered based on the curricular content as well as the appropriateness/usefulness of the construct in
determining a participant’s understanding of the topical content covered, their level of confidence and/or need
for additional support services in the case of the psychological constructs.
For the TRAMCON Intake survey, there are additional demographic survey items that address a participants’
level of construction experience, family involvement in the construction industry and the existence of
supporting individuals in a participant’s life (i.e., mentors and role models). Figures 2, 3 and 4, respectively,
provide a visual representation of the KSAO constructs included in the TRAMCON Intake, Foundation Level PreTest and Foundation Level Post-Test instruments. The survey items included in the TRAMCON Intake,
Foundation Level Pre-Test and Foundation Level Post-Test instruments can be found in Appendices F, G and H,
respectively. Surveys for the Basic, Advanced and Supervisory Training follow the same framework as the
Foundation level training but are tailored to the content-specific knowledge items appropriate for each
respective level of TRAMCON training.
Figure 2: KSAO Constructs included in the TRAMCON Intake Survey
Colleges Linking Data:
1) Unique Student ID Number Assigned at Intake
2) Date of birth
3) College for Linking
USDOL
1) Current Employment
2) Wage Rate
3) Other Items required by USDOL to show success
Demographic Data (CSU is Providing)
1) Construction Exp. Hands-On
2) Construction Exp. Management
3) Family Involvement in Construction
4) Mentors and Role Models
Knowledge, Skills, Abilities (KSA) Data
1) Test for Adult Basis Education TABE M9 (MDC)
(Note: NCCER, MSSC, MC Level 1 knowledge items will be
included in the training level pre- and post-tests)
Other Data
1) Construction Training Attitudes and Intention Scale (CTAIS)
- Self-Efficacy
- Motivation
- Planned Training Behavior
2) Grit Scale
- Resilience
3) Learning Styles Inventory
- Identifies Student Learning Styles
Figure 3: KSAO Constructs includes the TRAMCON Foundation Pre-Test Survey
Colleges Linking Data:
1) Unique Student ID Number Assigned at Intake
2) Student Date of Birth
3) Student College Attending
Knowledge, Skills, Abilities (KSA) Data
1) NCCER CORE (Selection of Modules 1, 2, 3, 4, 5, 7, 8, 9)
2) MSSC Knowledge Items (Selection Module 1, 2, 3, 4)
3) MC Level 1 (Selection of all content)
Figure 4: KSAO Constructs includes the TRAMCON Foundation Post-Test Survey
Colleges Linking Data:
1) Unique Student ID Number Assigned at Intake
2) Student Date of Birth
3) Student College Attending
Knowledge, Skills, Abilities (KSA) Data
1) NCCER CORE (Selection of Modules 1, 2, 3, 4, 5, 7, 8, 9)
2) MSSC Knowledge Items (Selection Module 1, 2, 3, 4)
3) MC Level 1 (Selection of all content)
Other Data
1) Construction Training Attitudes and Intention Scale (CTAIS)
- Self-Efficacy
- Motivation
- Planned Training Behavior
2) Grit Scale
- Resilience
* The CTAIS and GRIT survey items are also included in the
intake survey (that is, it serves as a pre-test) for these
constructs. Since these dynamic/changing constructs may be
impacted by the completion of the training (the
intervention), we will investigate increases in student selfefficacy etc. after the foundation level is completed.
KSAO Policy
All consortium member institutions will use the agreed upon procedures to assess and collect data. All students
will participate in the assessments and sign an acknowledgement that they agree to participate in the
administration of these instruments.
KSAO Assessment Administration
The KSAO assessments (Intake and all pre- and post-tests for each level of training) can be administered to
TRAMCON participants in electronic (internet-based) or hardcopy form. In general, the TRAMCON consortium
uses Qualtrics® survey software to generate the internet-based surveys for administration. The advantage of
electronic administration of the survey (via software such as Qualtrics®) is that response data is automatically
complied in an electronic format for use by the TRAMCON colleges and data collection specialists. There are
instances where internet access is unavailable or intermittent and therefore hardcopy survey administration is
required. Additional data input steps are required when hardcopy KSAO assessments are administered. In the
case of TRAMCON, hardcopy surveys are read and hand-entered into the Qualtrics® interface. Hand-entered
data are checked by the individual entering the data and are then rechecked against the originals by another
individual to ensure data entry accuracy. Entering KSAO hardcopy data into the Qualtrics® software allows the
benefit of mass data export for input into the ETO interface. In addition, data checking insures that participant
data are not duplicated in the rare case where a participant completes both an internet-base and hardcopy
surveys for the same KSAO assessment.
KSAO Data Post Processing and Distribution to TRAMCON Colleges
The Qualtrics® software has been programmed to complete certain statistical algorithms prior to exporting the
data. For example, in the case of NCCER content knowledge items, a participant’s number of correct answers
and/or a percentage score on the assessment is the most appropriate/usable data point for TRAMCON
instructors and administrators. The Qualtrics® software produces these aggregated results prior to data
export. While TRAMCON instructors may gain access to participant responses for a specific survey item on the
KSAO (i.e., Question #5 on the NCCER content knowledge portion of the foundation pre-test), the data is
generally provided in its usable aggregated form. Once data is exported to Microsoft Excel®, it is cleaned and
screened for duplicate responses, incomplete surveys and outliers. The data is separated for each participating
college and uploaded to a file sharing software (i.e., DropBox®) for distribution. Data for each college are
provided to the respective college only and the full data set for all colleges is imported to the ETO interface.
Using the TRAMCON Foundation pre- and post-test as an example, the following data are provided to the
TRAMCON colleges. While data for pre- and post-tests are administered at the beginning (pre-) and end (post)
of the Foundation Level Training, the data for both assessments are provided in one spreadsheet for
comparison. An example of the output data spreadsheet for the Foundation Level Training is provided in
Appendix I. Table 2 provides a visual representation of the data collected at Intake, pre- and post-Foundation
Level Training. The psychological data (e.g., “Other”) obtained via the TRAMCON intake survey is the pre-test
and the post-test for these constructs occurs at the end of the Foundation Level Training (the intervention).
Table 2: TRAMCON Pre- and Post-Test Data Comparison, Intake through Foundation Post-Test
KSAO
Constru
ct
Other
Other
Other
TRAMCON Intake Survey
Foundation Level Pre-Test
Foundation Level Post-Test
-
Manufactured Construction
Training Self-Efficacy (Post-Test)
Manufactured Construction
Training Self-Efficacy (PreTest)
Manufactured Construction
Training Motivation (Pre-Test)
Manufactured Construction
Training Planned Behavior
(Pre-Test)
-
Manufactured Construction
Training Motivation (Post-Test)
Manufactured Construction
Training Planned Behavior
(Post-Test)
Other
Grit/Resilience (Pre-Test)
-
Grit/Resilience (Post-Test)
Other
Learning Style Inventory (PreTest ONLY)
-
-
NCCER Content Knowledge
(Pre-Test)
(NCCER CORE Modules 1, 2,
3, 4, 5, 7, 8, 9)
MSSC Knowledge Items
(Pre-Test)
(MSSC CPT Modules 1, 2, 3,
4)
NCCER Content Knowledge
(Post-Test)
(NCCER CORE Modules 1, 2, 3,
4, 5, 7, 8, 9)
Knowle
dge
-
Knowle
dge
-
Knowle
dge
-
MC Level 1 (Pre-Test)
MSSC Knowledge Items (PostTest)
(MSSC CPT Modules 1, 2, 3, 4)
MC Level 1 (Post-Test)
KSAO Data Use and Ethics
The KSAO assessment and data are designed to assist TRAMCON practitioners in quantitative measurement of
the Knowledge, Skills, Abilities and Other (i.e. psychological) characteristics shown in occupational and
educational research to predict training behaviors including performance and drop out. The KSAO assessment
should be used in a manner that positively serves trainees. TRAMCON participant response data should be
used to align training provider student services with individual TRAMCON participant needs. Better alignment
of trainee needs and training services through interpretation of TRAMCON KSAO assessments are intended to
increase trainee performance and completion rates and decrease attrition. The KSAO assessments are not
intended to be a benchmarking tool to identify and/or select training participants. Further, the KSAO
assessments should not be used to exclude a particular individual (i.e. someone with low self-efficacy) from
participation in the TRAMCON training program.
References:
Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50(2),
179-211.
Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, NJ:
Prentice Hall.
Oddessa College Student Success Center (N.D.). Learning Style Inventory. Retrieved 4-9-2016 from:
http://www.sjsu.edu/eop/students/workshops/ACADEMIC_Learning%20Style%20Inventory.pdf
Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. (2007). Grit: Perseverance and passion for longterm goals, Journal of Personality and Social Psychology, 92(6), 1087–1101. DOI: 10.1037/0022-3514.92.6.1087
Eagle, M. N. (2011). Theories of motivation. In G. Gabbard, B. Litowitz & P. Williams (Eds.), Textbook of
psychoanalysis (2nd ed., pp. 39-52). Arlington, VA: American Psychiatric Publishing.
Elliott, J. W., &Lopez del Puerto, C. (2015). Development of an attitudes and intention scale for construction
skills training programs. Journal of Employment Counseling, 52(3), 131-144. DOI: 10.1002/joec.12011