Utilizing an Outcomes-Based
Program Review to Inform Resource
Allocations
Lauren Weiner, Ed.D.
Director, Associated Students Administration
University of California-San Diego
Marilee J. Bresciani, Ph.D.
Professor, Postsecondary Educational Leadership
San Diego State University
Ben Gillig, M.A. Candidate
Assistant Coordinator of Academic Initiatives
San Diego State University
Lisa McCully, M.A.
Director, College of Education Office of Student Services
San Diego State University
Bresciani, Gillig, Weiner, & McCully
Session Overview
 Presentation of framing questions
 Review of research study
 Discussion of findings and application
to practice
 Presentation of a preliminary
framework
 Discussion with small groups
Bresciani, Gillig, Weiner, & McCully
Framing Questions
 What evidence is used to inform your
resource re-allocation or allocation
processes?
 How does your outcomes-based program
review process inform resource re-allocation
or allocation process?
 Why would you even want to have this
conversation?
Bresciani, Gillig, Weiner, & McCully
Background
 Resource allocation is about more than just money…
Human resources
 Facilities
 IT
 Time spent by senior leaders
 Budget allocations
 Evidence that priorities are being effectively met
 Resource allocation and re-allocation are cornerstones of
senior student affairs officers’ jobs
 How resources are allocated says something about how
the division operates and how it sets its priorities

Bresciani, Gillig, Weiner, & McCully
Student Affairs Leaders Pulling The Pieces Together
Programs
&
Services
Divisional Budget Allocation
Human Resources
Donations
Capital
Inputs
Bresciani, Gillig, Weiner, & McCully
Outputs
Resource
Allocation
Literature Review
Budgeting Frameworks
Program budgeting
 Formula budgeting
 Zero-base budgeting
 Performance budgeting
 Incremental budgeting

Require a set of
purposes,
priorities,
and/or
strategies to
work.
What framework do we use to ensure that our purposes,
priorities, and/or strategies will be effective in accomplishing
intended results?
Bresciani, Gillig, Weiner, & McCully
Purpose of the Study
 This mixed method study explores how
institutional/divisional leadership use outcomes-based
assessment results to inform resource allocations and reallocations for student learning and development.
 Explores the need to utilize outcomes-based assessment in
times of budget cuts, reduction in revenues, and a higher
demand for services
 The findings intend to identify frameworks that align
resource allocations and re-allocations with stated
priorities and provide evidence to refine programs and
services that align with those priorities.
Bresciani, Gillig, Weiner, & McCully
Hypotheses
 The manner in which institutions are funded does
not influence how institutional leadership uses
outcomes-based assessment results to inform
resource allocations and re-allocations for student
learning and development.
 The timeframe in which institutional leadership
allocates and re-allocates resources for student
learning and development does not influence how
they use outcomes-based assessment results to
inform the allocation and re-allocation of resources
for student learning and development.
Bresciani, Gillig, Weiner, & McCully
Hypotheses, Cont.
 The budgeting framework that institutional leadership uses
to allocate and re-allocate resources for student learning
and development does not influence how they use
outcomes-based assessment results to inform the allocation
and re-allocation of resources for student learning and
development.
 The manner in which institutional leadership engages in
strategic planning does not influence how they use
outcomes-based assessment results to inform the allocation
and re-allocation of resources for student learning and
development.
Bresciani, Gillig, Weiner, & McCully
Hypotheses, Cont.
 The manner in which institutional leadership
engages in outcomes-based assessment program
review does not influence how they use outcomesbased assessment results to inform the resource
allocation and re-allocation of resources for
student learning and development.
Bresciani, Gillig, Weiner, & McCully
Methodology
 This mixed methods study included the following data




collection processes:
Survey list started with institutional members (n=1,030)
Researchers stratified the institution by type
A random sample of 257 invitations were then sent through
Student Voice.
Survey invitations were sent to the institution’s Senior
Student Affairs Officer
 E.g., VP of Student Affairs or Student Services
Bresciani, Gillig, Weiner, & McCully
Methodology
 After data was collected, researchers analyzed the
information using descriptive statistics
 Researchers looked for correlations between
institutional/divisional budget process and how they
utilized their outcomes-based assessment results for their
resource allocations and re-allocations
 Researchers conducted a document analysis based on the
information that was provided from their institutions URL
addresses
 Interviews will be conducted in the upcoming weeks
Bresciani, Gillig, Weiner, & McCully
Survey Participant Demographics
 Institution type
Private 4-year Liberal Arts Colleges (35%)
Public 4-year Comprehensive (20%)
Public 2-year (15%)
 Institution size
Less than 4,000 (37%)
4,000 to 8,000 students (24%)
15,001 to 25,000 students (22%)
 Position title
Senior Student Affairs Officer (93%)
Bresciani, Gillig, Weiner, & McCully
Instrument Development
• Survey included 25 questions and a request for institutions
to provide their URL address for additional assessment
documents
 Survey focused on asking institutional leaders to discuss if
they utilize (a) outcomes-based assessment, (b) program
review, (c) annual reports, and (d) strategic planning to
inform the budget and resource allocation and re-allocation
process at the institutional and divisional level
 Survey included questions pertaining to
institutional/divisional leadership practices and questions
relating to the practices used for allocating and reallocating resources to student learning and development
programs
Bresciani, Gillig, Weiner, & McCully
Document Analysis
• Survey respondents were asked to provide their URL addresses
• Document analysis selection criteria:
• All materials needed to be official institutional documents, which
•
•
•
•
•
•
included:
Annual reporting processes
Outcomes-based assessment processes
Program review processes
Assessment plans from the institutional, divisional, and departmental
levels
Researchers analyzed documents and developed themes by using the
comparative method for category development
Researchers determined essential descriptive groupings during the
initial coding phase
Bresciani, Gillig, Weiner, & McCully
Instrument Development
Interview Protocol:
• Researchers will conduct semi-structured interviews with institutional
leaders, such as vice-presidents, deans, faculty, and administrators
directly involved in the outcomes-based assessment program review
process
• Researchers will also conduct interviews with institutional leaders
involved in the resource allocation and re-allocation processes
Questions focus on:
• Types of evidence based decision-making processes that institutional
leadership use to inform the resource allocation and re-allocation
process for student learning and development
• The alignment between the results generated from outcomes-based
program review and resource re-allocations and allocations
Bresciani, Gillig, Weiner, & McCully
Limitations
 Since the researchers used a self-reporting survey,
participants did not have to provide their actual
budgeting documents, thus researchers cannot
triangulate that data; however, outcomes-based
and planning practices can be triangulated through
document analysis
 25% response rate; survey is back out in the field
Bresciani, Gillig, Weiner, & McCully
Preliminary Findings
Bresciani, Gillig, Weiner, & McCully
The manner in which institutions are funded does not
influence how institutional leadership uses outcomesbased assessment results to inform resource allocations
and re-allocations for student learning and
development.
W E D O N O T H AV E C O N C L U S I V E D ATA F O R
REJECTING THIS HYPOTHESIS
Bresciani, Gillig, Weiner, & McCully
There were significant correlations between certain revenue sources and use of
outcomes based assessment at certain points in the resource allocation/reallocation process.
Use of tuition and fee revenue was associated with use of OBA to formulate
budget requests at both the institutional and division levels but not to inform
actual resource allocation (t=-.43, p>.01; t=-.48, p>.01).
Use of state allocated revenue was associated with OBA playing a role in reallocation only (t=-.34, p=.02).
Use of state allocation revenue was also associated with OBA playing a
weighted role in the formulas used to allocate money to departments (t=.83,
p=.01).
Use of endowment revenue was associated with OBA playing a weighted role
in the formulas used to allocate money to divisions (t=.31, p=.04).
Use of revenue from auxiliary services was associated with OBA playing no
role in resource allocation/re-allocation at all (t=-.32, p=.02).
Based on these preliminary findings, we will be conducting interviews and
gathering additional documents to clarify
Bresciani, Gillig, Weiner, & McCully
The timeframe in which institutional leadership allocates and re-allocates resources for
student learning and development does not influence how they use outcomes-based assessment
results to inform the allocation and re-allocation of resources for student learning and
development.
• 98% of survey participants responded that
their resource allocation process takes place
on an annual basis
• Since the majority of respondents selected
the same timeframe, we can not determine
the extent that the time frame influences
this process.
• Thus, we can not accept or reject this
hypothesis
Bresciani, Gillig, Weiner, & McCully
The budgeting framework that institutional leadership uses to allocate and re-allocate resources for
student learning and development does not influence how they use outcomes-based assessment
results to inform the allocation and re-allocation of resources for student learning and
development.
 46% report their institutions using incremental
budgeting processes, while 35% use programbased at the divisional level
 There are no significant correlations between
budgeting framework and the use of OBA for
resource allocation and re-allocation
 We accept this hypothesis
Bresciani, Gillig, Weiner, & McCully
The manner in which institutional leadership engages in strategic planning does not
influence how they use outcomes-based assessment results to inform the allocation
and re-allocation of resources for student learning and development.
• Strategic planning is the most common practice used to inform the
•
•
•
•
•
•
resource allocation or re-allocation process
Institution level: 22% of survey respondents reported that strategic
planning informs the resource re-allocation process
Institution level: 20% of survey respondents reported that strategic
planning played a role in the way they construct the budget, but not the
allocation of resources
Division level: 22% of survey respondents reported that strategic planning
informs the resource re-allocation process
Division level: 22% of survey respondents also reported that strategic
planning played a weighted role in the formulas used to allocate dollars
There are no significant correlations between the use of strategic planning
and the use of OBA for resource allocation and re-allocation
We accept this hypothesis.
Bresciani, Gillig, Weiner, & McCully
The manner in which institutional leadership engages in outcomes-based program
review does not influence how they use outcomes-based assessment results to
inform the resource allocation and re-allocation of resources for student learning
and development.
• Institution level: 24% of survey respondents reported that program
review does not have any role in the allocation or re-allocation of
resources
• Institution level: 20% of survey respondents reported that program
review plays a role in the re-allocation of resources
• Division level: 27% of survey respondents reported that program
review plays a role in the re-allocation of resources
• Division level: 22% of survey respondents reported that program
review does not play any role in the allocation or re-allocation of
resources
Bresciani, Gillig, Weiner, & McCully
The manner in which institutional leadership engages in outcomes-based program
review does not influence how they use outcomes-based assessment results to
inform the resource allocation and re-allocation of resources for student learning
and development.
• However, uses of outcomes-based program review data was correlated
with several uses of outcomes-based assessment data:
•
•
•
•
Use of program review data in a weighted role in budget formulas used to allocate money to
departments was correlated with the use of outcomes based assessment data for the same purpose
(t=.43, p<.01).
Use of program review data in a weighted role in the budget formula used to allocate money to
programs and processes correlated with the use of outcomes based assessment data for the same
purpose (t=.47, p<.01).
Use of program review data to construct budget proposals, but not necessarily to allocate resources
was correlated with the use of outcomes based assessment data for the same purpose (t=.67,
p<.001).
Use of program review data to construct budget proposals, but definitely not to allocate resources
was correlated with the use of outcomes based assessment data for the same purpose (t=.69,
p<.001).
• Thus, we cannot reject this hypothesis.
Bresciani, Gillig, Weiner, & McCully
Preliminary Findings from the Document
Analysis
 Documents analyzed from the URL addresses included annual


•
•
reports, assessment plans, program reviews, strategic
planning processes, assessment committee structures, and
institutional planning documents
Themes from the documents:
Departmental goals and objectives align with the student
affairs and institutional goals and objectives
Institutions use assessment results to inform the program
improvement process, not in the resource allocation process
Identification of departmental and divisional successes, but
do not inform the resource allocation process
Bresciani, Gillig, Weiner, & McCully
Similarities and Differences Between Survey and
Documents
• Institutional leaders identified and communicated values,





objectives, and priorities
Departments aligned programs with the institutional goals
and objectives
Input being solicited from faculty, staff, and administrators
Strong participation of faculty, staff, and administrators in the
planning process
Differences between survey and documents:
Survey respondents indicated that resource re-allocations
were influenced by strategic planning and outcomes-based
assessment program review findings; documents showed no
linkage to the resource re-allocation process
Bresciani, Gillig, Weiner, & McCully
Summary of Conclusions
•
•
•
There is a disconnect between what revenue sources
influence versus what the budgeting process influences.
There is a disconnect between the institutional budgeting
processes and the ability to use allocation of resources to
improve student learning and development. Does that make
sense to continue?
If 74% of the initial allocation of funding is done annually,
why aren’t outcomes-based program review results used to
inform initial allocations of resources?
Bresciani, Gillig, Weiner, & McCully
Summary of Conclusions, Cont.
•
If strategic planning and outcomes-based assessment
program review findings only influence resource reallocations and not initial allocations, how will leadership be
able to influence long-term improvement in student
learning and development?
•
More to come….
Bresciani, Gillig, Weiner, & McCully
Framework with Modifications
(Bresciani, 2010 & Bresciani, et.al, in progress)
 Identify and articulate values
 Prioritize values
 Allocate resources
 Align outcomes to values and Implement outcomes-
based assessment
 Define the criteria for quality within the context of the
values and identify capacity for meeting the criteria of
quality
 Gather the results and determine at which level the
decision for resource re-allocation or allocation resides
Bresciani, Gillig, Weiner, & McCully
Framework with Modifications
(Bresciani, 2010 & Bresciani, et.al, in progress)
 Allocate or re-allocate resources to improve your
outcomes within your context and capacity for quality
and in alignment with your values
Bresciani, Gillig, Weiner, & McCully
Small Group Discussion Questions
 How well do your resource budgeting and allocation
processes align?
 How much do you utilize planning and assessment
processes to allocate or re-allocate resources to
refine your priorities?
 Does this proposed framework offer any practical
value for your division/institution?
 What are some immediate next steps that you can
implement?
Bresciani, Gillig, Weiner, & McCully
Planned Next Steps for the Study
 Complete correlation analysis
 Re-invite participants to complete the
survey
 Request more URLs for expanded analysis particularly budget process documents
 Conduct interviews to test the viability of the
framework
 Write-up results for publication submission
Bresciani, Gillig, Weiner, & McCully
Special Thanks to…
 Student Voice
 JD
White
 Annemieke Rice
 Participating institutions
 To you
Bresciani, Gillig, Weiner, & McCully
Session Citation
 This presentation:
 Weiner, L., Bresciani, M.J., Gillig, B., McCully, L. (2011). Using
outcomes-based assessment to inform resource allocation in
higher education. Workshop presented at the annual meeting
of the American College Personnel Association, Baltimore,
MD.
 Future paper:
 Bresciani, M.J., Weiner, L., Gillig, B., McCully, L.(in
preparation). Using outcomes-based assessment to inform
resource allocation in higher education. Journal of Student
Affairs Research and Practice submission
Bresciani, Gillig, Weiner, & McCully
References
Bresciani, M.J. (2010). Aligning values with resources and assessment
results. Student affairs leader (38), 13, p. 1-2.
Bresciani, M. J., Gardner, M. M., & Hickmott, J. (2009). Demonstrating
student success in student affairs. Sterling, VA: Stylus Publishing.
Berg, D.J. & Skogley, G.M. (1985). Editors’ notes. In D.J. Berg & G.M.
Skogley (Eds.), Making the budget process work. New Directions for
Higher Education No. 52. San Francisco: Jossey-Bass.
Carnegie Council on Policy Studies in Higher Education (1980). Three
thousand futures: The next twenty years for higher education. San
Francisco: Jossey-Bass.
Clark, R. & d’Ambrosio’s, M. (2006). The new balancing act in the
business of higher education. Northamton, MA: Edward Elgar Books.
Bresciani, Gillig, Weiner, & McCully
References, Cont.
Merriam, S. B. (1998). Qualitative research and case study application in
education. San Francisco: Jossey-Bass.
Mayhew, L. (1979). Surviving the eighties: Strategies and priorities for
solving fiscal and enrollment problems. San Francisco: Jossey-Bass.
Strauss, A., & Corbin, J. (1990). Basics of qualitative research. Newbury
Park, CA: Sage.
Taylor, S. J., & Bogdan, R. (1998). Introduction to qualitative research
methods. New York: John Wiley.
Bresciani, Gillig, Weiner, & McCully
Contact Information
 Marilee J. Bresciani –
[email protected]
 Ben Gillig –[email protected]
 Lauren Weiner – [email protected]
 Lisa McCully – [email protected]
Bresciani, Gillig, Weiner, & McCully