Evaluation of Sector Strategies:
What We Know Matters to States
Impact on Jobseekers and Workers
•
•
•
•
Increased availability of good jobs
Improved working conditions
Expanded work supports
Increased opportunities for education and
training for high-demand occupations
• Increased employment and wage gains
Impact on Employers and Industry
• Shared costs and risks
• Increased availability of skills training
• Guidance on ways to improve human
resource practices
• Reduced turnover
Effectiveness of the Partnership
Systems Change
• The right partners “at the table”
• Industry challenges identified
• Appropriate solutions
designed/implemented
• Plan of action, road map, goals and
outcomes
• Partners agree the Partnership is valuable
• New and leveraged funding
•
•
•
•
Changes in how education, workforce,
economic development and employers
work together
New social and business supports
Changes in public policy
Changes in employer practices
Two Notes: 1) NGA White Paper and Evaluation Framework offers full detailed discussion on these categories;
2) a 5th category of growing interest is “impact on community” (such as reduced poverty/unemployment)
1
Is There One Formula for Evaluation?
First, can we fairly compare
outcomes? Consider . . .
Second, do we assume training
outcomes? Consider . . .
• Over 1,000 healthcare service
workers trained every year by a
metropolitan sector partnership
• About 130 students per year
trained and placed in
employment by a solar
installation sector partnership
• 3 Journeyman apprentices
supplied to Grand Coulee Dam as
a result of a Power Generation
partnership
• A small manufacturing sector
partnership that focuses on
career awareness
• Or an energy partnership that
develops skills standards (that
can be used to develop training)
but that primarily is valuable to
align titles and HR selection
criteria across employers
No single formula, but there are
templates to capture some consistent
data across partnerships.
2
From Washington State
From Pennsylvania
PA INDUSTRY PARTNERSHIP ANNUAL REPORT 2008-2009*
Overview: Pennsylvania’s Workforce Development system is increasingly focused on promoting Industry
Partnerships (IPs), a key institutional innovation for meeting the skills needs of businesses, the career goals
of workers and the economic development goals of the commonwealth. To help improve the effectiveness of
IPs, the Department of Labor and Industry requires each IP to complete an Annual Report, also known as
the High Performance Standards for Industry Partnerships.
Category of Impact
Describe Activity
Incumbent Worker Training
e.g. new/revised curriculum or new credentials based
on industry needs
Organizational Effectiveness (of
the workplace/business)
e.g. mentorships, management training, career
pathways, support services, case management
Building the Pipeline
e.g. career awareness, job shadowing, internships,
outreach to special populations
Influencing Regional Institutions
e.g. changes by LWIB, one-stops, post-secondary, k12, business associations, labor orgs, CBOs, econ. dev.
Governance, Collaboration &
Sustainability (of the IP)
e.g. core competencies such as industry analysis,
capacity building, planning, coordinating,
implementing, self-evaluating their business impact
Impact?
Annual Report also includes requests for 1) testimonials; 2) greatest success; and 3) greatest challenge
*Note: Adapted and condensed from 7-page 08-09 Annual Report Template – PA
From Massachusetts
Annual Report August 2009*
Purpose: To provide an opportunity for each project to reflect on its progress toward meeting its goals and share
what is learned with Commonwealth Corporation. These reports offer a means to learn more about how projects are
unfolding - both at the level of individual projects and across the WCTF initiative.
Category of Performance
Type of Information Requested
Part I: Quarterly Update
Describe activities during last Qtr, successes, innovations, challenges, next steps
Part II: Program Goals (of
jobseeker or worker participants)
# participants; # un-/underemployed; # incumbent; #
completers; # new credentials; # wage increase; other
Part III: Outreach, Recruitment
and Selection (of participants)
Describe any successes/challenges, changes you plan to make within each area.
On scale of 1-5, rate your success with each.
Part IV: Training
How are providers selected? How do they participate? Please provide your
assessment of training to date. What changes will you make?
Part V: Participant Supports
What types of supports do your participants need (paid release time, coaching,
case management, tutoring, childcare, etc)? Describe challenges and changes.
Part VI: Partnership
List partners, their roles and indicate in which types of activities they participate.
Describe benefits for partners, challenges, future partner contributions.
Part VII & VIII: Employer
Engagement and Biz Impact
How are employers involved? Is engagement low, medium or high? Using the
“Getting Started” plan, what baseline and impact data do you have?
Part IX: Products of Project
Describe progress on contractual deliverables. Describe other products such as
new curriculum, new certificate program, new partnerships, others.
*Note: Adapted and condensed from 10-page WCTF – Annual Report, Round Two, Year 1, August 2009
On Track? Y/N
Will reach goal when?
Employer Engagement
[Source: WCTF Final Report 2009]
•
To help us understand the nature of employer engagement
in your project, please complete the following table. For
each activity listed, check () the share of employers (all,
most, some/a few, none) who engaged in the activity.
–
–
–
–
–
–
–
–
–
–
–
Attend regular partnership meetings
Provide guidance on the overall direction of the project
Contribute to financial match
Participate in on-going strategic planning for sustain-ability
Recruit Participants
Screen and select participants
Participate in training provider selection
Provide input on training content/ curriculum
Participate in training provision
Interview, hire or offer internships to participants
Other (Please specify):
Systemic Change
[Source: WCTF Final Report 2009]
Please describe any systemic changes that have occurred as a result of your
project. Focus on the three areas described here as appropriate. Your
responses may include, but need not be limited to, changes such as the
examples provided in each area.
1) Education, Training, Social and Business Supports
–
new, leveraged and/or redeployed resources that improve service delivery
–
changes in practice or policy within educational institutions
–
improved responsiveness to employers from public institutions
2) Employer Practices
–
–
–
new HR practices
new career ladders
new/leveraged private resources
3) Public Policy
–
–
new/leveraged funding from legislature
industries engage with the public sector to address workforce needs
Some Evaluation Resources*
Evaluations of Sector Initiatives
Targeting Industries, Training Workers, and Improving Opportunities: Final Report of the Sectoral Employment
Initiative, Public-Private Ventures, 2008: http://www.ppv.org/ppv/publications/assets/263_publication.pdf
BEST (Building Essential Skills through Training) Benefits: Employer Perspectives, Volume 2, Issue 4, Research and Evaluation Brief,
Commonwealth Corporation, 2004: http://www.commcorp.org/researchandevaluation/pdf/ResearchBrief2-04.pdf
Benefits of a Sector-Based Approach, Volume 2, Issue 3, Research and Evaluation Brief, Commonwealth Corporation, 2004:
http://www.commcorp.org/researchandevaluation/pdf/ResearchBrief2-03.pdf
Performance and Evaluation Models for Sector Initiatives
An Evaluation Framework for State Sector Strategies, product of 11-state project of the National Governors Association,
The Corporation for a Skilled Workforce and the National Network of Sector Partnerships, 2008:
http://www.sectorstrategies.org/system/files/EvaluationFrameworkWhitePaper.pdf
Evaluating Industry Skill Panels: A Model Framework, Commissioned by the Washington State Workforce Education and
Training Coordinating Board, Corporation for a Skilled Workforce with the Paros Group, June 2008:
http://www.wtb.wa.gov/Documents/2008SkillPanelEvaluationReport.pdf
* All available at www.sectorstrategies.org
Field of Evaluation is Ever- Evolving
• Since March ‘09, six states in this project (MI, MA, PA, WA, IL, WI) convened four
times via conference call to discuss key issues in sector strategy evaluation
• Short (but rich) summaries of each call are on the blog at
www.sectorstrategies.org (have you registered yet?)
• Topics for discussion have included:
– “Systems Change” can be a nebulous, confusing term; there’s a need to de-mystify; concrete
indicators of systems change already exist within our repertoire of sector evaluation, including
career ladder development, changed HR protocols, new certificate/degree programs, others.
– A “lessons learned” discussion about measuring impact on employers from MA’s experience in
the healthcare/long-term care industry; Common data points: turnover, vacancy rates,
reductions in agency fees – i.e. for firm to contract out for skilled work –, and overtime costs.
Lesson learned: must collect qualitative testimony in addition to quantitative because latter not
always available, and certainly not quickly.
– Measurement Consistency across sector partnerships: WA takes the “choose 3” approach; not
perfect. Also need common definitions for indicators (such as reduced cost of turnover). PA drew
heavily from the NGA projects framework to develop the 2008-2009 annual report. Two other
issues: how do we get honest results? How do we account for differences across industries?
– PA put a healthcare impact report together summarizing efforts and outcomes of 13 healthcare
partnerships – this is potentially a powerful way to capture impact more specifically.
Any Questions?