Self-Assessment of MTSS (SAM) Technical Assistance Manual July 2016 Stockslager, K., Castillo, J., Brundage, A., Childs, K., & Romer, N. A collaborative document between Florida’s Problem Solving/Response to Intervention Project and Florida’s Positive Behavior Intervention and Support Project, discretionary projects of the Florida Department of Education and the University of South Florida. Please cite as: Stockslager, K., Castillo, J., Brundage, A., Childs, K., & Romer, N. (2016). Self-Assessment of MTSS (SAM) Technical Assistance Manual. Florida’s Problem Solving/Response to Intervention Project and Florida’s Positive Behavior Intervention and Support Project, University of South Florida. Table of Contents SAM Technical Assistance Manual Introduction .............................................................................................................i Description and Purpose ................................................................................................................................................. 1 Intended Audience .......................................................................................................................................................... 1 Training Required ........................................................................................................................................................... 1 Directions for Administration ......................................................................................................................................... 2 Scoring and Analysis ...................................................................................................................................................... 2 Frequency of Use ............................................................................................................................................................ 3 Technical Adequacy Initial Pilot ........................................................................................................................................................ 3 National Pilot .................................................................................................................................................... 4 Interpretation and Use of the Data .................................................................................................................................. 8 School-Level Example of SAM Data ............................................................................................................................ 10 Appendix A: Copy of the Self-Assessment of MTSS (SAM) ......................................................................................... 13 Appendix B: Additional Considerations for SAM Collection, Analysis, and Interpretation ........................................ 35 Appendix C: Standardized Factor Loading for Items on the SAM ............................................................................... 40 SAM Technical Assistance Manual Introduction Program evaluation of Multi-Tiered Systems of Supports (MTSS) initiatives is a critical component of facilitating successful implementation. Complex educational systems require that key stakeholders take a systems view of facilitating change and develop plans to address variables likely to relate to successful implementation. Educators’ knowledge and skills; school, district, and state policies and procedures; funding streams; and myriad other factors likely will impact whether educators will adopt practices within an MTSS framework. Although a comprehensive strategic plan designed to address these systemic factors is a necessary condition for successful implementation, it is not sufficient by itself. Formative data-based evaluation of needs within the educational system and the impact of any actions taken should be used to guide the development of (and modifications to) implementation plans. Key stakeholders who engage in this type of formative decision-making can focus more intensely on identified issues, thus responding to the specific needs of educators and the systems in which they operate. The development of a model to evaluate efforts to scale up MTSS implementation, however, poses several challenges. Questions about what issues to focus on and how often to collect data, among others, can be difficult to address. It is with these difficulties in mind that the evaluation team from the Florida Problem Solving/Response to Intervention (PS/RtI) Project and Florida’s Positive Behavior Intervention and Support Project created this SelfAssessment of MTSS (SAM) technical assistance manual. The purpose of this manual is to provide information about the SAM to educational stakeholders interested in using the instrument to inform MTSS implementation. A summary of the information available on the SAM follows. • • • • • • • • • • i Description & Purpose of the Instrument: Theoretical background, description of the instrument, and its intended use Intended Audience: Suggestions for who should complete the instrument and who should use the results for decision-making Training Required: Suggestions for training of individuals responsible for (1) administering or completing the instrument and (2) analyzing and interpreting the results Directions for Administration: Strategies for administering or completing the instrument and examples of ways in which Project staff approached administration Scoring and Analysis: Strategies for summarizing data for decision making Frequency of Use: Considerations when determining how often to use the instrument and general guidelines for frequency of use Technical Adequacy: Available information on the reliability and validity of the instrument Interpretation and Use of the Data: Suggestions for analyzing, displaying, and interpreting results School-Level Example of SAM Data: Examples of how data could be collected, displayed, and used to guide decisions made at the school-level Appendices: A copy of the instrument and additional information relevant to SAM administration, analysis, and interpretation Description & Purpose Description The Self-Assessment of MTSS (SAM) is a needs assessment and progress-monitoring tool for implementation of a multi-tiered system of supports (MTSS). A copy of the instrument is found in Appendix A, page 13. Data from the SAM can be used to inform implementation efforts, in conjunction with school and district improvement plans and other implementation data. The SAM is a 39-item self-report measure organized around six domains associated with implementation of an MTSS model. The six domains assess the extent to which schools are (1) developing leadership for implementation, (2) developing the capacity and infrastructure necessary to support implementation, (3) building communication and collaboration structures, (4) engaging in data-based problem solving, (5) implementing a three-tiered instruction/intervention model, and (6) building a comprehensive data and evaluation system. Each item within these domains is scored using a rubric with the following response options: 0= Not Implementing 1= Emerging/Developing 2= Operationalizing 3= Optimizing Purpose The purpose of the instrument is to assess current implementation levels of an MTSS model to inform schools and districts regarding which areas require action planning. The SAM can assist educators in identifying areas of need in their MTSS and monitoring implementation progress. Intended Audience Who Should Complete the SAM? School-Based Leadership Team (SBLT) members complete the SAM. SBLTs are comprised of approximately six to eight staff members selected to take a leadership role in facilitating MTSS implementation in a school. (Note. For more information on the development of SBLTs, see Appendix B, page 35). Who Should Use the Results for Decision Making? The SBLTs who complete the SAM should review the results for their school. The District-Based Leadership Team (DBLT) should review the results for the district’s schools individually, as well as aggregated across schools. Members of the DBLT provide leadership and support to schools implementing practices within an MTSS. Training Required Training Recommended for Individuals Facilitating SAM Completion Qualifications of the Facilitator. Completion of the SAM requires that an individual be in charge of facilitating SBLT consensus regarding item ratings. Personnel in charge of facilitating completion of the SAM should have a thorough understanding of the MTSS model and the systems issues that influence implementation. Facilitators also should possess the interpersonal and communication skills required to facilitate consensus among a group of individuals that may have different opinions regarding the extent to which the school is engaging in certain MTSS activities. Content of the Training. Trainings on facilitating completion of the SAM should include the following components: • • • • • Explanation of the relationship between implementation integrity and desired outcomes, and the alignment between the SAM and critical features of implementing practices within an MTSS. Review of each domain and item so that facilitators have a clear understanding of what is being measured. Description of the SAM Endnotes and how team members can use Endnotes to enhance understanding. Overview of administration and scoring procedures. Common issues that arise during administration, such as frequently asked questions and how to address disagreements among team members. Training Suggested for Analyzing, Interpreting, and Disseminating SAM Results. The knowledge, skills, and experience of educators in analyzing, interpreting, and using data for decision making may vary. If the stakeholders responsible for these activities possess the knowledge and skills required, then training specific to the SAM data may 1 not be necessary. However, should the stakeholders responsible for using the data lack any of the aforementioned skill sets, training and technical assistance is recommended. Topics that support may include: • • • Appropriate use of the measure, given its purpose and technical adequacy. Analyzing and displaying data derived from the tool. Interpreting and disseminating the results. Directions for Administration The SAM is completed by SBLT members in three steps: Step 1. The facilitator reviews the SAM with SBLT members to ensure their understanding of the purpose of the SAM, what the instrument measures, how the information will be used, and how to complete the SAM. Step 2. The facilitator provides each SBLT member a copy of the SAM to review individually approximately one week prior to the meeting at which the team reaches consensus on a single score for each item. Disseminating the instrument before this SBLT meeting provides adequate time for participants to record their perspectives and to attend the meeting ready to contribute to discussion. Step 3. The facilitator guides discussion until the team reaches consensus on a score for each item. The facilitator then records the final responses. Group completion of the SAM typically takes one to two hours, depending on the amount of discussion required to reach consensus on each item. Use the SAM version that represents the consensus of the SBLT members for decision-making purposes. Note. After completion of the SAM, SBLT members can meet to review the results, identify specific areas of need, and identify potential action items for improving MTSS implementation. (Note. For more information on SAM administration, see Appendix B, page 35). Scoring and Analysis The amount of analysis required to use the SAM for decision making will likely depend on the unit of analysis (e.g., school, district, state). School-Level Analysis of SAM Data School-level personnel using the results simply may want to chart responses from the school’s final version to identify needs and monitor progress over time (see example school-level graphs below in Figures 1, 2, and 3 – pages 10, 11, and 12). District-Level Analysis of SAM Data Stakeholders examining other units of analysis (e.g., district-level, schools served across a state or geographic region) likely would need to aggregate results to inform decision making. Two ways in which personnel aggregating results from multiple schools can consider analyzing data from the SAM include (1) calculating the mean rating for each domain and item to determine the average activity level evident across schools and (2) calculating the frequency of each response option selected (i.e., Not Implementing, Emerging/Developing, Operationalizing, Optimizing) for each item. Calculating domain and item means provides an overall impression of the MTSS implementation activities occurring within each of the six SAM domains. When calculating average implementation levels, the following values should correspond with each response option: 0 = Not Implementing; 1 = Emerging/Developing; 2 = Operationalizing; 3 = Optimizing. Examining implementation at the domain level allows educators to examine general patterns across each of the six SAM domains. One can compute a domain score by calculating the sum of the ratings of the items that comprise the domain and dividing by the total number of items within the domain (see example district-level “average domain scores” graph in Appendix B; Figure 4, page 36). The items that comprise the six domains are as follows: • • • • Domain 1 (Leadership): Items 1-5 Domain 2 (Building Capacity/Infrastructure): Items 6-16 Domain 3 (Communication and Collaboration): Items 17-20 Domain 4 (Data-Based Problem Solving): Items 21-27 2 • • Domain 5 (Three-Tiered Instructional/Intervention Model): Items 28-33 Domain 6 (Data and Evaluation): Items 34-39 Calculating the mean rating for each item across schools allows stakeholders to identify the extent to which educators are engaging in specific activities to facilitate MTSS implementation. This information can be used to identify specific activities that may need to be addressed systematically (through professional development, policies and procedures, etc.), but does not provide detailed information regarding the variability across schools for each activity. Calculating the frequency of schools reporting levels of implementation for an item (Not Implementing, Emerging/Developing, Operationalizing, and Optimizing), on the other hand, provides information on the range of activity levels. This information helps to determine the percentage of schools engaged in specific MTSS implementation activities. When making decisions about how to address implementation efforts, information on the number of schools engaging in a particular activity can help inform decisions regarding modifying implementation plans (see example district-level “item frequency” graphs in Appendix B; Figure 8, page 39). Technology Support School personnel should consider using district supported or commercially available technology resources1 (e.g., SurveyMonkey®, Google Forms®, Qualtrics®) to facilitate collection and analyses of the data. Frequency of Use When determining how often SBLT members should complete the SAM, it is important to consider the resources available within schools and districts to support data collection. Important considerations include: • • • • • The time needed for completion of the instrument The time required to enter, analyze, graph, and disseminate data The personnel available to support data collection Other data collection activities in which SBLT members and school staff are required to participate The time required to increase implementation of specific practices within an MTSS. In other words, decisions about how often to collect SAM data should be made based on the capacity to administer, analyze, and use the information to inform plans to scale-up MTSS implementation. Although schools and districts will need to make adjustments given available resources, general recommendations for completing the SAM are as follows. Completing the SAM once per year can assist SBLT members in identifying implementation levels of specific practices. SBLTs can use the information obtained to develop short- and long-term goals for implementing MTSS as well as to develop strategic and action plans (e.g., professional development activities and necessary supports). Administering the SAM again each year will allow SBLT members to examine progress made during the previous year and to refine goals and action plans for the subsequent school year. Technical Adequacy Initial Pilot. The initial pilot phase of the SAM began in 2013 and included several actions designed to address the content validity of the SAM. Those actions are described below and resulted in a 39-item version of the SAM used for data collection during the national pilot. Content Validity The content validation process of the SAM involved several steps. First, there was a thorough review of related research (e.g., MTSS, problem solving, RtI, PBIS, educational systems change) to identify critical components associated with implementation of MTSS. Additionally, there was a review of existing implementation evaluation instruments to identify strengths and areas for improvement in the field of MTSS implementation evaluation. After the initial items were generated, Florida PS/RtI Project and Florida PBIS:MTSS Project staff provided feedback regarding the importance and clarity of each item, resulting in a revised item list. Next, an Expert Review Panel, which consisted of 11 district-, state-, and national-level experts on MTSS (RtI) and/or PBIS implementation reviewed the draft instrument. The panel members provided feedback on the 1 These are included to provide stakeholders with examples of technology resources available to assist with data collection. The Project does not endorse specific technology resources. 3 representativeness of the MTSS components covered by the items using the following scale: Not at all relevant, Somewhat relevant, Relevant, Very relevant, and Don’t know. Panel members also provided feedback on the clarity and conciseness of the individual items using the following scale: Not at all, Somewhat, Very, Don’t know. Finally, panel members suggested modifications to items using open-ended response options. The Projects’ evaluation staff analyzed panel member feedback and revised the survey using a structured process. Evaluation staff considered 80% agreement among panel members that an item was relevant and well written as the criterion for retaining an item in its current form. When agreement from the panel members was below 80%, the staff reviewed and discussed feedback from the respondents who disagreed with the item. A review of panel results indicated that 97% of items met the criterion for agreement that the content was relevant and that 74% of the items met the criterion for clarity. Items that did not meet the 80% criteria for content relevancy or item clarity were revised based on qualitative feedback by reviewers. Once panel members’ comments were addressed, each revised item was compared to panel member feedback to calculate a revised estimate of agreement that the item was relevant and clear. This process resulted in all 34 items meeting the 80% criterion for retaining individual items. Following the expert panel review process, cognitive interviews were conducted with six school-level stakeholders (e.g., school administrator, student support personnel, content specialist, teacher). The purposes of the cognitive interviews were to (1) solicit feedback on the clarity of the SAM items and (2) ensure that stakeholders were accurately interpreting the items by having interviewees verbalize their thought process for each item. For each item, interviewees thought aloud while reading and provided feedback on clarity and interpretation. Each of the six interviewees also provided specific feedback on any terms or phrases that could be confusing or be considered jargon. Information gathered during the interviews was used to revise items identified as problematic. Following the cognitive interviews, a small-scale pilot administration was conducted in 155 schools across seven districts from two states. The 155 SBLTs completed the 34-item version of the SAM that emerged from the content development stage described above and provided qualitative feedback on item groups (e.g., additional components to be included in the SAM). Data collected from this administration and feedback from the SBLTs resulted in the addition of five items to the original 34-item version of the SAM (i.e., revisions resulted in a 39-item version). National Pilot. Following the activities completed during the initial pilot study, a large scale, national pilot study was conducted to address the construct validity and reliability of the SAM. Construct Validity Confirmatory factor analysis (CFA) procedures using a categorical model were used to examine the 6-factor structure of the SAM that was conceptualized from the literature. Data from SAMs completed by 436 SBLTs from 15 districts within eight states were used to analyze the instrument. The fit for the model was examined using Bentler’s (1992) comparative fit index (CFI) and the root mean square error of approximation (RMSEA; Steiger & Lind, 1980). CFI values greater than or equal to .95 and RMSEA values less than or equal to .08 (Hu & Bentler, 1999) were considered to indicate acceptable levels of fit. The model estimated resulted in good fit (CFI = .96, RMSEA = .05). Standardized loadings ranged from .69 to .83 for items on the Leadership factor (5 items), from .68 to .85 for the Building Capacity/Infrastructure factor (11 items), from .66 to .84 for the Communication and Collaboration factor (4 items), from .62 to .85 for the Data-Based Problem Solving factor (7 items), from .79 to .91 for the Three-Tiered Instructional/Intervention Model factor (6 items), and from .79 to .87 for the Data and Evaluation factor (6 items; see Table 4 in Appendix C, page 40, for a listing of the individual item loadings). Correlations between each of the factors were high (see Table 1 below for correlations between each of the factors). 4 Table 1. Correlations between SAM Domains. Leadership Leadership Capacity/ Infrastructure Communication and Collaboration Data-Based Problem Solving Three-Tiered Model Data and Evaluation Capacity/ Infrastructure Communication Data-Based and Problem Collaboration Solving Three-Tiered Model Data and Evaluation * .86 * .79 .85 * .79 .85 .87 * .69 .75 .73 .86 * .83 .90 .86 .88 .80 * Internal Consistency Reliability Internal consistency reliability estimates were computed for each of the six domains using Cronbach’s alpha. Cronbach’s alphas ranged from .79 to .91 indicating adequate to high levels of internal consistency. Specific Cronbach’s alpha coefficients for each of the factors were: • • • • • • Leadership: α = .84 Building Capacity/Infrastructure: α = .91 Communication and Collaboration: α = .79 Data-Based Problem-Solving: α = .89 Three-Tiered Instructional/Intervention: α = .90 Data and Evaluation: α = .90. Criterion Validity. The relationships between the SAM and the Benchmarks of Quality (BoQ; Kincaid, Childs, & George, 2005) were examined using data from 188 schools that also completed the BoQ. A review of the Pearson correlation coefficients showed a moderate relationship between the BoQ total score and the overall SAM score. Small to moderate correlations also were found between the BoQ total score and several domain scores from the SAM. See Table 2 for correlations between the SAM and BoQ. Table 2. Correlations between the SAM and the Benchmarks of Quality (BoQ). SAM Overall Leadership Capacity/Infrastructure Communication and Collaboration Data-Based Problem Solving Three-Tiered Model Data and Evaluation * p < .05. ** p < .01 *** p< .001. BoQ .31*** .28*** .28*** .12 .32*** .35*** .15* The relationships between the SAM and several behavior outcomes also were examined in a subset of schools who provided behavior outcome data. The behavior outcomes examined included the number of Office Discipline Referrals (ODRs), the number of Out of School Suspension Events (OSS-Events), and the number of 5 Out of School Suspension Days (OSS-Days). The total score from the SAM was negatively associated with the number of OSS days. Three of the six domain scores also were negatively associated with OSS days. Finally, Leadership was negatively related to the number of ODRs. The magnitudes of the relationships found were small. See Table 3 for correlations between the SAM and the behavioral outcomes investigated. Table 3. Correlations between the SAM and Behavioral Outcomes. SAM Overall Leadership Capacity/Infrastructure Communication and Collaboration Data-Based Problem Solving Three-Tiered Model Data and Evaluation * p < .05. ** p < .01 *** p< .001. ODRs (n = 258) -.05 -.12* -.04 OSS-Events (n = 255) -.08 -.06 -.07 OSS-Days (n = 243) -.14* -.10 -.13* -.04 -.05 -.13 -.02 -.06 -.09 -.02 -.06 -.05 -.12 -.13* -.16* Relationships between the SAM and academic outcomes also were examined. The academic outcome data available included Florida Standards Assessment2 (FSA) data for 261 Florida schools. We examined correlations between the percentage of students proficient on the FSA for the English Language Arts and Mathematics subtests, and SAM scores from their schools. See Tables 4 through 8 for correlations between the SAM and the academic outcomes for all schools in the sample, elementary, middle, high, and secondary (middle and high) schools, respectively. Results differed by content area for the entire sample of schools. Both the total score and five of the six domain scores were positively associated with the percentage of students who were proficient on the Mathematics subtest. Only the Three-Tiered Instruction/Intervention implementation score was positively associated with the percentage of students proficient on the English Language Arts subtest. The magnitudes of the correlations were small. Results differed somewhat when broken down by school level. The Data-Based Problem Solving and ThreeTiered Instruction/Intervention scores were positively associated with proficiency on both the English Language Arts and Mathematics subtests at the elementary level. The magnitudes of the correlations were small. No other correlations were significant at the elementary level. At the high school level, only the Leadership domain scores were associated with proficiency on the FSA. Specifically, Leadership scores were positively associated with English Language Arts and Mathematics proficiency (the magnitudes of the correlations were moderate). No other scores were associated with FSA proficiency at the middle school, high school, or combined secondary levels. 2 For more information on the Florida Standards Assessments (FSA), see http://www.fldoe.org/accountability/assessments/k-12-student-assessment/fsa.stml 6 Table 4. Correlations between the SAM and Academic Outcomes (All Schools). SAM Overall Leadership Capacity/Infrastructure Communication and Collaboration Data-Based Problem Solving Three-Tiered Model Data and Evaluation * p < .05. ** p < .01 *** p< .001. English Language Arts (% Proficient) (n = 261) .10 .07 .04 Mathematics (% Proficient) (n = 261) .19** .12 .14* .08 .16* .12 .21*** .13* .10 .23*** .15* Table 5. Correlations between the SAM and Academic Outcomes (Elementary). SAM Overall Leadership Capacity/Infrastructure Communication and Collaboration Data-Based Problem Solving Three-Tiered Model Data and Evaluation * p < .05. ** p < .01 *** p< .001. English Language Arts (% Proficient) (n = 173) .12 .05 .05 Mathematics (% Proficient) (n = 173) .13 .06 .06 .09 .11 .16* .18* .16* .12 .18* .11 Table 6. Correlations between the SAM and Academic Outcomes (Middle). SAM Overall Leadership Capacity/Infrastructure Communication and Collaboration Data-Based Problem Solving Three-Tiered Model Data and Evaluation * p < .05. ** p < .01 *** p< .001. English Language Arts (% Proficient) (n = 45) .16 .05 .10 Mathematics (% Proficient) (n = 45) .13 -.02 .08 .16 .15 .11 .09 .26 .14 .25 .14 7 Table 7. Correlations between the SAM and Academic Outcomes (High). SAM Overall Leadership Capacity/Infrastructure Communication and Collaboration Data-Based Problem Solving Three-Tiered Model Data and Evaluation * p < .05. ** p < .01 *** p< .001. English Language Arts (% Proficient) (n = 37) .14 .33* .13 Mathematics (% Proficient) (n = 37) .23 .37* .24 .09 .21 .10 .14 .02 .04 .04 .17 Table 8. Correlations between the SAM and Academic Outcomes (Secondary). SAM Overall Leadership Capacity/Infrastructure Communication and Collaboration Data-Based Problem Solving Three-Tiered Model Data and Evaluation * p < .05. ** p < .01 *** p< .001. English Language Arts (% Proficient) (n = 82) .15 .17 .12 Mathematics (% Proficient) (n = 82) .16 .12 .14 .12 .12 .11 .11 .16 .10 .19 .13 Interpretation and Use of the Data Broad domains to examine When interpreting SAM data, examine the six broad domains measured by the instrument (i.e., Leadership, Building Capacity/Infrastructure, Communication and Collaboration, Data-Based Problem Solving, Three-Tiered Instructional/Intervention Model, Data and Evaluation) first. Key stakeholders (e.g., SBLTs, DBLTs) can examine graphically displayed data to evaluate levels of implementation within each of the domains. Each of the methodologies for scoring mentioned above (i.e., calculating average activity levels at the domain and item levels and calculating the frequency/percent of schools who selected each response option at the item level) can be used to examine the broad domains. (Note. For more information on interpretation and use of SAM data, see Appendix B, page 35). Identifying specific needs Each item within the domains also can be graphed to examine trends in which activities tend to be engaged in more or less frequently. Key stakeholders should consider a number of factors when identifying which activities tend to be engaged in at relatively high levels versus those being engaged in at low levels. For example, schools may need to address leadership structures and processes before focusing on capacity building and implementation. 8 Sharing data with stakeholders It is important that a plan for disseminating data on implementation and for providing key stakeholders the time and support to discuss the information be included in a plan to scale-up practices within an MTSS. It is recommended that these key stakeholders be identified and data be shared with them as quickly and frequently as possible following SAM administration. This timeline allows stakeholders, such as SBLT members, to discuss activity levels indicated by the SAM data, develop or alter goals, and design strategies (e.g., professional development, access technology resources, develop procedures) to facilitate increased levels of implementation. DBLT members also should have access to data from schools to plan support provided at the district level. Additionally, SBLT and DBLT members may find it helpful to have a coach or facilitator discuss the data with stakeholders to facilitate interpretation and problem-solve barriers to implementation efforts. Finally, SBLT members are highly encouraged to share their data with instructional staff members. These stakeholders often are critical to the implementation of practices within an MTSS and their support and input are important to consider when developing and finalizing action plans. Using questions to guide discussions To facilitate discussions about implementation efforts, one helpful strategy is to provide educators with guiding questions. The use of guiding questions is designed to facilitate discussions about each school’s data, including potential strategies for increasing the use of practices within an MTSS. Listed below are examples of guiding questions used to facilitate discussions regarding implementation. These guiding questions were designed to facilitate discussions about each school’s data, including current level of problem-solving implementation and consistency between SAM data and other implementation measures. • • • What domains have the highest and lowest levels of implementation? What specific items represent the highest and lowest levels of MTSS implementation? Which specific MTSS implementation actions or activities will your team focus on improving within your school or district? - Which are most immediately actionable? - Which would be most impactful? - Which would be most foundational (e.g., aligned with where you want your school or district to be regarding MTSS implementation)? 9 School-Level Example of SAM Data SunshineElementary:SAM AverageDomainScores 3 2.5 2 1.5 1 0.5 0 Figure 1. School-Level Example of SAM Data: Average Domain Scores. Explanation of the Graph Sunshine Elementary recently committed to implementing MTSS at the school. The newly formed SBLT at Sunshine Elementary met at the beginning of the school year to plan for implementation, but realized that they did not know where to begin. At the suggestion of the school’s MTSS coach, the team decided to complete the SAM at their next meeting to inform goals and activities for the year. They also agreed to complete the instrument again at the end of the year to examine progress and identify additional needs. After completion of the SAM, the team first reviewed the average domain scores to identify potential areas of strengths and need (Figure 1). A quick review of the graph led the team to identify Leadership as an area to focus on related to improvement of MTSS implementation. While the team noted low levels of implementation in several domains (e.g., data-based problem solving, three-tiered model), the team also understood the importance of leadership to improving implementation. To better understand specific areas for improvement within the Leadership domain, the school leadership team decided to review individual item responses within the domain. 10 Figure 2. School-Level Example of SAM Data: Leadership Domain. Explanation of the Graph As mentioned above, the Sunshine SBLT decided to first focus on the Leadership domain after a review of their SAM average domain scores. Figure 2 above includes results for the items from the SAM that assess leadership activities. The results are from the beginning of the year administration. Interpretation and Use of the Data Examining items within a SAM domain. Following the administration of the SAM at the beginning of the year, the SBLT met to discuss the results and to plan for addressing leadership levels. First, the SBLT took note of the initial status of leadership activities reflected by the SAM items displayed in Figure 2. Team members noted that principal involvement (item 1), establishment of an SBLT (item 2), and having an MTSS implementation plan (item 4) were all “emerging/developing.” They also noted that the school had not started engaging staff in professional development and coaching (item 3) or actively facilitating implementation of MTSS (item 5). Overall, these data indicated that work needed to be done to build leadership for MTSS implementation at the school before schoolwide implementation could occur. SBLT members proceeded to plan for how to increase leadership activities at the school. Identifying specific needs. Given that the SBLT noted that the school had not started or was “emerging/developing” for leadership activities, they decided that soliciting more active principal involvement and further defining the roles and expectations of the SBLT should be the initial focus. SBLT members met with school administrators to discuss the development of a vision for MTSS implementation, the critical role of administrators in the process, and how to identify professional development and support required to build capacity for implementation, as well as to establish regular communication with the administrators. Additionally, the SBLT – with support of the administration decided to improve their own functioning (item 2) by creating opportunities to involve additional staff members, developing expectations for facilitating MTSS implementation, and engaging in activities to build their beliefs, knowledge and skills regarding implementation. SBLT members established regular team meetings, developed roles 11 and expectations for the team, and planned for internal professional development to increase their own knowledge and skills. After the SBLT addressed team functioning issues, the SBLT began to identify specific activities to facilitate MTSS implementation as part of their ongoing school improvement process (item 5). They provided ongoing professional development to staff, based on data from a school-wide needs assessment and in alignment with their school improvement plan, and developed a strategic plan. Figure 3. School-Level Example of SAM Data: Leadership Domain (BOY = Beginning of Year, EOY = End of Year). Examining trends in SAM data. After developing roles and expectations for the SBLT and engaging in internal professional development efforts, Sunshine Elementary was interested in how their school’s leadership activities changed throughout the year. See Figure 3 to examine the beginning and end of year SAM results. The red bars, representing the end of year SAM data, demonstrated increases in indicators of leadership development for most items. Specifically, the school had “operationalized” the establishment of a functioning SBLT with explicit expectations for facilitating MTSS implementation. The SBLT also engaged in additional activities throughout the year to build on their internal progress. For example, while the SBLT noted that engaging staff in professional development and coaching for MTSS implementation (item 3) was not present at the beginning of the year, the activity was “emerging/developing” by the end of the school year. While the comparison of beginning of year to end of year data shows promising changes for Sunshine Elementary, it is critical to remember that leadership development is an ongoing activity. The SBLT members agreed to continue to monitor staff beliefs, knowledge, and skills in order to ensure that the proper supports were being provided to increase levels of MTSS implementation. 12 Appendix A. Copy of the Self-Assessment of MTSS (SAM) Self-AssessmentofMTSSImplementation(SAM) OverviewofMTSS Thisinstrumentisusedtomeasureschool-levelimplementationofaMulti-TieredSystemof Supports (MTSS). MTSS is a term used to describe an evidence-based model of educating students that uses data and problem solving to integrate academic, behavior, and socialemotionalinstructionandinterventiontomaximizethesuccessofallstudents.Instruction and intervention is provided to students across multiple tiers of intensity based on need. Staffmakedata-baseddecisionsinorderforresources(e.g.,time,staff,andevidence-based strategies) to reach the students at the appropriate levels to increase the performance of ALLstudentswiththegoalofachievingand/orexceedingproficiency. Quality implementation of MTSS is associated with increased likelihood of instruction and interventions leading to successful student outcomes. Thus, it is important for schools to monitor not only student outcomes, but also how assessments, instruction, interventions, and data-based problem-solving are put into place (i.e., the fidelity with which these elements are implemented). Successful implementation is influenced by many factors within and around the school system (e.g., professional development, administrative support,datasystems,staffmemberperceptions,successfuladaptation,etc.).Asameasure ofschool-levelimplementationofanMTSS,thefocusofthisinstrumentisonthenecessary actions and activities to successfully implement and sustain the critical elements of MTSS withfidelity.ThecriticalelementsofMTSSreferredtothroughouttheinstrumentinclude: • Curriculumstandards • Assessmentsusedtoinforminstruction • Multipletiersofinstructionandintervention • Data-basedproblem-solvingusedtomakedecisions To promote a common understanding, staff that complete the instrument are urged to discusstheelementsofMTSSandhowtheyrelatetocomponentsoftheirschool’ssystem for educating all students. MTSS should not be thought of as a separate initiative or programthatmustbeimplemented.Rather,MTSSprovidesaframeworkfortheintegration of academic, behavior, and social-emotional supports. Other initiatives such as implementation of educational policies and regulations, new assessment systems, or new instructional strategies also should be considered in the context of how they fit withinan MTSS. MTSS provides a framework for implementing educational practices to ensure academic,behavioral,andsocial-emotionalsuccessofallstudents. SAMVersion2.0(LastrevisedOctober2015)Page-i 13 DirectionsforCompletingtheInstrumentandUsingtheData The school leadership team that has responsibility for allocating resources to improve student learning should complete this instrument. Completion involves a three-step process: 1. Each team member should review the SAM instrument and Endnotes independentlyandthinkhows/he,personally,wouldrespondtoeachitem. 2. After reviewing the SAM items independently, the team members should come together to discuss their responses and reach agreement on which answer best represents the current status of implementation at their school.Endnotesprovideadditionalclarifyinginformationordefinitionsthat the team should utilize, especially as team members are first becoming familiar with the SAM instrument. Endnotes provide critical information for ensuring the SAM instrument is completed accurately and results in valid scores.RecordconsensusscoresontheSAMScoringSheet.TheScoringSheet hasabbreviatedlanguageandshouldonlybeusedtorecordresponsesandto provideavisualrepresentationofitemswithhigherandlowerscores. 3. UsetheSAMinstrumentandtheScoringSheetdatatoinformyouractionplan (anoptionalplanningtemplateisprovided)toimproveMTSSimplementation. Rate each item on a scale from 0-3 (0 = Not Started; 1 = Emerging/Developing; 2 = Operationalizing;3=Optimizing)usingthedefinitionprovidedforeachrating. Superscript numbers (e.g., 2) correspond with endnotes that provide additional clarifying informationordefinitionsrelevanttothecontentwithintheitem. Thereare39itemsorganizedintosixdomains: 1) Leadership Leadership is key to successful implementation of any large-scale innovation. The building principal, assistant principal(s), and school leadership team are critical to implementing MTSS at the school level. They engage staff in ongoing professional developmentforimplementingMTSS,planstrategicallyforMTSSimplementation,and model a data-based problem-solving process for school improvement. The school principal also supports the implementation of MTSS by communicating a vision and mission to school staff, providing resources for planning and implementing instruction andintervention,andensuringthatstaffhavethedataneededfordata-basedproblemsolving. SAMVersion2.0(LastrevisedOctober2015)Page-ii 14 2) BuildingtheCapacity/InfrastructureforImplementation School-widecapacityandinfrastructurearerequiredinordertoimplementandsustain MTSS. This capacity and infrastructure usually includes ongoing professional developmentandcoachingwithanemphasisondata-basedproblem-solvingandmultitieredinstructionandintervention;schedulingthatallowsstafftoplanandimplement instructionandintervention;andprocessesandproceduresforengagingindata-based problem-solving. 3) CommunicationandCollaboration Ongoing communication and collaboration are essential for successful implementation of MTSS. Many innovations fail due to a lack of consensus, to a lack of feedback to implementerstosupportcontinuousimprovement,andtonotinvolvingstakeholdersin planning. In addition to including stakeholders in planning and providing continuous feedback,itisalsoimportanttobuildtheinfrastructuretocommunicateandworkwith families and other community partners. These practices increase the likelihood that innovativepracticeswillbeimplementedandsustained. 4) Data-BasedProblem-Solving The use of data-based problem-solving to make educational decisions is a critical elementofMTSSimplementation.Thisincludestheuseofdata-basedproblem-solving forstudentoutcomesacrosscontentareas,gradelevels,andtiers,aswellastheuseof problem-solving to address barriers to school wide implementation of MTSS. While several models for data-based problem-solving exist, the four-step problem-solving approachevaluatedinthisinstrumentincludes:1)definingthegoalsandobjectivesto be attained, 2) identifying possible reasons why the desired goals are not being attained,3)developingaplanforandimplementingevidence-basedstrategiestoattain thegoals,and4)evaluatingtheeffectivenessoftheplan. 5) Three-TieredInstructional/InterventionModel The three-tiered instructional/intervention model is another critical element of MTSS implementation. In a typical system, Tier 1 includes the instruction delivered to all students;Tier2includessupplementalinstructionorinterventionprovidedtostudents not meeting benchmarks; and Tier 3 includes intensive, small-group or individual interventions for students facing significant barriers to learning the skills required for school success. It is important to consider academic, behavior, and social-emotional instructionandinterventionswhenexaminingthisdomain. 6) Data-Evaluation Given the importance of data-based problem-solving within an MTSS model, the need for a data and evaluation system is clear. In order to do data-based problem-solving, school staff need to understand and have access to data sources that align with the purposes of assessment. Procedures and protocols for administering assessments and data use allow school staff to use student data to make educational decisions. In addition to student data, data on the fidelity of MTSS implementation allow school leadership to examine the current practices and make changes to increase implementation. SAMVersion2.0(LastrevisedOctober2015)Page-iii 15 Acknowledgements The Florida Problem Solving/Response to Intervention (PS/RtI) Project and Florida’s Positive Behavior Support: MTSS (FLPBS: MTSS) Project would like to thank the following groups for theireffortsonthisinstrument: FloridaPS/RtIProjectstaff Florida’sPBS:MTSSProjectstaff MembersoftheSAMExpertReviewPanel ParticipantsintheSAMCognitiveInterviewProcess SomeitemsontheSAMwereadaptedfromthe RtIImplementationRubric:SchoolLevelbytheColoradoDepartmentofEducation Pleaseciteas: Stockslager,K.,Castillo,J.,Brundage,A.,Childs,K.,&Romer,N.(2016).Self-Assessmentof MTSS(SAM).Florida’sProblemSolving/ResponsetoInterventionProjectandFlorida’s PositiveBehaviorInterventionandSupportProject,UniversityofSouthFlorida SAMVersion2.0(LastrevisedOctober2015)Page-iv 16 Item 3=Optimizing andTheleadershipteamprovides supporttoeducatorsimplementing thecriticalelementsofMTSS identifiedinthestrategicplan andAspartoftheschool improvementplanningprocessa strategicplanisdevelopedthat 4 specifiesMTSSimplementation andAprofessionaldevelopment planiscreatedbasedontheneeds assessmentandusedtoengage staffinongoingprofessional developmentandcoaching Rating andTheleadershipteamusesdata onimplementationfidelityofthe criticalelementsofMTSStoengage indata-basedproblem-solvingfor thepurposeofcontinuousschool improvement andAstrategicplanforMTSS implementationisupdatedas neededbasedonstudentoutcome andimplementationfidelitydataas partoftheschoolimprovement planningprocess andOngoingprofessional developmentactivitiesare informedbydatacollectedonthe outcomesofprofessional developmentandcoachingfor continuousimprovement andTheleadershipteamhas andTheleadershipteammembers explicitexpectationsforfacilitating havethebeliefs,knowledge,and MTSSimplementation, skillstoleadimplementationefforts SAMVersion2.0(LastrevisedOctober2015)Page-1 Theleadershipteamisnot activelyengagingineffortsto facilitateMTSS implementation Theleadershipteamengagesin actionplanning andhascreatedastrategicplan tofacilitateimplementationof 6 thecriticalelements ofMTSS Aneedsassessmentisconducted togatherinformationonbeliefs, knowledge,andskillstodevelop aprofessionaldevelopmentplan tosupportMTSSimplementation 5.Theleadershipteamis activelyfacilitating 5 implementationofMTSS as partoftheirschool improvementplanning process Theleadershipteamdoes nothaveaneeds-basedplan toprovidestaffwith professionaldevelopmentor coachingtosupportMTSS implementation 3.Theleadershipteam activelyengagesstaffin ongoingprofessional 3 developmentandcoaching necessarytosupportMTSS implementation Aleadershipteamexiststhat includescross-disciplinary representation, Leadershipteamisengaging district,family,andcommunity partnerstoidentifystakeholder needs,resourcesfor,andbarriers toMTSSimplementation Noleadershipteamwith explicitresponsibilityfor leadingMTSS implementationexists 2.Aleadershipteamis establishedthatincludes6-8 memberswithcrossdisciplinaryrepresentation (e.g.,principal,generaland specialeducationteachers, contentareaexperts, instructionalsupportstaff, 1 studentsupportpersonnel ) andisresponsiblefor facilitatingMTSS 2 implementation 2=Operationalizing Theprincipalcommunicatesan andTheprincipalactivelysupports andThePrincipalactivelysupports urgentdesiretoimplement theleadershipteamandstaffto data-basedproblem-solvinguseat MTSS,participatesinprofessional buildcapacityforimplementation theschool developmentonMTSS,andis establishinganMTSSvision 4.AstrategicplanforMTSS NostrategicplanforMTSS implementationisdeveloped implementationexists andalignedwiththeschool improvementplan Theprincipaldoesnot activelysupportMTSS. 1.Theprincipalisactively involvedinandfacilitates MTSSimplementation 1=Emerging/Developing Self-AssessmentofMTSSImplementation(SAM) 0=NotImplementing 1.LeadershipDomain(Items1-5) 17 18 Item 6 1=Emerging/Developing Noinformationonthe criticalelementsofthe school’sMTSSisavailable Thestaffengagesininitial,jobembeddedprofessional developmentfocusingon: • Purposeandadministrationof assessmenttools • Roleofassessment/data sourcesinmakinginstructional decisions • Reviewofcurrent assessments/datasources beingutilizedandthosebeing considered • Analyzingandusing assessmentresultstoimprove instruction • Usingvarioustypesofdatato informinstructionalpractices tomeettheneedsofdiverse learners • Communicatingandpartnering withfamiliesaboutdataand assessmentpractices ThecriticalelementsofMTSSare beingdefined 3=Optimizing Rating andThestaffengagesinongoing professionaldevelopmentand coachingrelatedtothe administrationofassessmentsand interpretationofthedata/data sources.Professionaldevelopment includes: • Changesorupdatesto assessments/datasources • Changestodatacollection, tracking,andanalysis • Ongoingcoachingon instructionalpracticesand interpretingassessmentresults andTheleadershipteamanalyzes feedbackfromstaffaswellas outcomesinordertoidentify professionaldevelopmentand coachingneedsintheareaof assessment/datasourcesinsupport ofcontinuousimprovement andThecriticalelementsofMTSS andThecurriculum,assessment, aredefinedandarecommunicated andinstructionalpracticesthat toschoolstaff definetheschool’scriticalelements ofMTSScanbecommunicatedby allschoolstaff SAMVersion2.0(LastrevisedOctober2015)Page-2 7.Theleadershipteam Initialprofessional facilitatesprofessional developmentisnotprovided 7 developmentandcoaching toallstaffmembers forallstaffmemberson assessmentsanddata sourcesusedtoinform decisions 6.Thecriticalelements of MTSSaredefinedand understoodbyschoolstaff 2=Operationalizing Self-AssessmentofMTSSImplementation(SAM) 0=NotImplementing 2.BuildingtheCapacity/InfrastructureforImplementationDomain(Items6-16) andOngoingprofessional developmentandcoachingon multi-tieredinstructionand interventionisprovidedthat includesthefollowingelements: • Differentiationofprofessional developmentandcoachingbased onstaffroles/responsibilities • Coaching • Modelingof,practiceof,and collaborativefeedbackon, evidence-basedpractices 2=Operationalizing andOngoingprofessional developmentandcoachingondatabasedproblem-solvingisdelivered thatincludesthefollowing elements: • Differentiationofprofessional developmentbasedonstaff roles/responsibilities • Coaching • Modeling,practice,and collaborativefeedbackon problem-solvingsteps • Supportforcollaborationand teamingskills SAMVersion2.0(LastrevisedOctober2015)Page-3 Initialprofessionaldevelopment onmulti-tieredinstructionand interventionisprovidedthat includesthefollowingelements: • Rationaleforandmodelingof instructionalandintervention designanddelivery(e.g., CommonCoreStateStandards, instructionalroutine,Tier1 PositiveBehaviorSupports, lessonplanningforactive studentengagement) • Connectionsaremade regardinghowthepractices arealignedwithand integratedintoMTSS • Howdatainformsinstruction andinterventiondesignand deliverythatreflectsstudent diversityandresultsinlearning 8 opportunitiesforallstudents 9.Theleadershipteam facilitatesprofessional 7 developmentandcoaching forallstaffonmulti-tiered instructionandintervention relativetotheirjob roles/responsibilities Noexplicitconnectionto multi-tieredinstructionand interventionisevidentin professionaldevelopment provided 1=Emerging/Developing Initialprofessionaldevelopment ondata-basedproblem-solvingis providedthatincludesthe followingelements: • Rationaleforuseofdata-based problem-solving • Problem-solvingstepsto addressschool-wide, classroom,small-group,and individualstudentneeds • Rolesandresponsibilitiesfor teammembersengagingin data-basedproblem-solving Self-AssessmentofMTSSImplementation(SAM) Item 0=NotImplementing 8.Theleadershipteam Professionaldevelopment facilitatesprofessional doesnotfocusondata-based 7 developmentandcoaching problem-solving forstaffmembersondatabasedproblem-solving relativetotheirjob roles/responsibilities andTheleadershipteamregularly usesdataonstudentneedsand fidelityofhowevidence-based practicesareimplementedto continuouslyimproveprofessional developmentandcoachingefforts Rating 3=Optimizing andDataonuseofproblem-solving skillsandapplicationareusedto informcontinuousimprovementof professionaldevelopmentand coachingefforts 19 20 Schedulesdonotinclude timeallocatedto professionaldevelopment andcoachingforMTSS Schedulesdonotinclude timeallocatedto administeringassessments neededtomakedecisions acrosstiers Themasterscheduleis developedwithoutstudent dataanddoesnotinclude timeformulti-tiered interventions 11.Schedulesprovide adequatetimefortrainings andcoachingsupport 12.Schedulesprovide adequatetimetoadminister academic,behaviorand social-emotionalassessments neededtomakedata-based decisions 13.Schedulesprovide adequatetimeformultiple tiersofevidence-based instructionandintervention tooccur 2=Operationalizing andCoachingactivitiesare expandedtoinclude: • Opportunitiestopractice • Collaborativeand performancefeedback andSchedulespermitpersonnelto administeradditionalassessment (e.g.,diagnosticassessments) acrosscontentareasandtiers neededtoengageindata-based problem-solving andSchedulespermitpersonnelto accessadditionaltrainingand coachingsupportthatis differentiatedbasedontheirneeds Rating 3=Optimizing andDataonprofessional development,implementation fidelity,andstudentoutcomesare usedtorefinecoachingactivities andThemasterschedulefacilitates andThemasterscheduleallowsfor effectiveimplementationofmulti- flexiblestudentgroupings tieredinterventionsmatchedto studentneedsbycontentareaand intensity(Tier1,Tier2,Tier3) andSchedulesincludetimeto administermorefrequentprogress monitoringassessmentsto studentsreceivingTier2and3 servicesasspecified(e.g.,weekly ormonthlyassessments) SAMVersion2.0(LastrevisedOctober2015)Page-4 Themasterscheduleis developedutilizingstudentdata andincludestimeformultitieredinterventions Schedulesincludetimefor academic,behaviorandsocialemotionalassessments administeredtoallstudents(e.g., universalscreening) Schedulesincludetimeallocated andSchedulesincludetimefor fortrainings ongoingcoachingsupport 1=Emerging/Developing Initialcoachingisoccurringthat isfocusedprimarilyon facilitatingormodelingthe componentsofMTSS Self-AssessmentofMTSSImplementation(SAM) 0=NotImplementing Nocoachingisprovidedto buildstaffcapacityto implementthecritical elementsofMTSS Item 7 10.Coaching isusedto supportMTSS implementation Leadershipteammembersare gatheringinformationonthe personnel,funding,materials, andotherresourcesavailableto supportMTSSimplementation Processes,procedures,and decision-rulesneededtoengage indata-basedproblem-solving aredevelopedandexisting structuresandresourcesare incorporated andResourceinventoriesare establishedusingthegathered informationonthepersonnel, funding,materials,andother resourcesavailabletosupport MTSSimplementation andplansforallocatingthe resourcesareestablished andThestepsofproblem-solving; proceduresforaccessing, submitting,andusingdata;and decision-rulesneededtomake reliabledecisionsare 10 communicatedtostaff 2=Operationalizing andThemasterscheduleprovides sufficienttimefortheprocessto occurwithfidelity SAMVersion2.0(LastrevisedOctober2015)Page-5 Noprocessexistsfor mappingandallocating resourcesavailableto supportMTSS implementation 16.Resources availableto supportMTSS implementationare identifiedandallocated 11 Nosystematicprocesses, procedures,ordecisionrulesareestablished 15.Processes,procedures, 9 anddecision-rules are establishedfordata-based problem-solving 1=Emerging/Developing Themasterscheduleprovides opportunitiestoengagein collaborative,data-based problem-solvinganddecisionmakingtooccur Self-AssessmentofMTSSImplementation(SAM) 0=NotImplementing Themasterscheduledoes notprovideopportunitiesfor collaborative,data-based problem-solvingand decision-makingtooccur Item 14.Schedulesprovide adequatetimeforstaffto engageincollaborative,databasedproblem-solvingand decision-making andExistingresourcemapsand resourceallocationsareupdatedat leastannuallybasedonstudent need,availablepersonnel,funding, materials,andotherresources andData-basedproblem-solving processes,procedures,and decision-rulesarerefinedbasedon dataandfeedbackfromstaff, schedulechanges,andresource availability Rating 3=Optimizing andThemasterscheduleprovides opportunitiesforcollaborative, data-basedproblem-solvingand decision-makingtooccurinsettings suchas: • Leadershipteammeetings • Grade-levelmeetings • Crossgrade-levelmeetings • Cross-departmentalmeetings • ProfessionalLearningCommunity meetings 21 22 Item 1=Emerging/Developing 12 Staffdo2or3ofthefollowing4: • activelyengagefamiliesthat representthediversepopulation oftheschool • engagefamiliesinproblem solvingwhentheirchildrenneed additionalsupports • provideintensiveoutreachto unresponsivefamilies • increasetheskillsoffamiliesto supporttheirchildren’s education Familyandcommunityengagement are2ofthefollowing3: • definedandmonitoredwithdata • linkedtoschoolgoalsin SIP/MTSSplan • supportedbyproceduresfor facilitating2-waycommunication exists Staffareregularly(2x/year) provideddataregardingMTSS implementationfidelity andstudentoutcomes andStaffhasopportunitiestogain understandingofitsrelevanceto theirrolesandresponsibilities SAMVersion2.0(LastrevisedOctober2015)Page-6 20.Educatorsactivelyengage Staffdononeofthe familiesinMTSS following: • activelyengagefamilies thatrepresentthediverse populationoftheschool • engagefamiliesin problemsolvingwhen theirchildrenneed additionalsupports • provideintensiveoutreach 16 tounresponsivefamilies • increasetheskillsof familiestosupporttheir children’seducation Staffdo1ofthefollowing4: • activelyengagefamiliesthat representthediverse populationoftheschool • engagefamiliesinproblem solvingwhentheirchildren needadditionalsupports • provideintensiveoutreachto unresponsivefamilies • increasetheskillsoffamiliesto supporttheirchildren’s education Familyandcommunity engagementare1ofthe following3: • definedandmonitoredwith data • linkedtoschoolgoalsin SIP/MTSSplan • supportedbyproceduresfor facilitating2-way communication Familyandcommunity engagementis:notdefined andmonitoredwithdata; notlinkedtoschoolgoalsin SIP/MTSSplan; andproceduresfor facilitating2-way communicationdonotexist 19.Theinfrastructureexiststo supporttheschool’sgoalsfor familyandcommunity 15 engagement inMTSS Staffareprovidedopportunities togainunderstandingofthe needforMTSS Staffarerarely(1x/year) provideddataregardingMTSS implementationfidelity andstudentoutcomes Staffarenotprovided opportunitiestogain understandingoftheneed forMTSS 18.Staffareprovideddataon Staffarenotprovidedany MTSSimplementationfidelity dataregardingMTSS 14 andstudentoutcomes implementationfidelitynor studentoutcomes 17.Staff haveconsensus andengageinMTSS 13 Implementation 2=Operationalizing Self-AssessmentofMTSSImplementation(SAM) 0=NotImplementing 3.CommunicationandCollaborationDomain(Items17-20) Rating Staffdoallofthefollowing: • activelyengagefamiliesthat representthediversepopulation oftheschool • engagefamiliesinproblem solvingwhentheirchildrenneed additionalsupports • provideintensiveoutreachto unresponsivefamilies • increasetheskillsoffamiliesto supporttheirchildren’s education Familyandcommunityengagement areallofthefollowing: • definedandmonitoredwithdata • linkedtoschoolgoalsin SIP/MTSSplan • supportedbyproceduresfor facilitating2-waycommunication exist Staffarefrequently(3x+/year) provideddataregardingMTSS implementationfidelity andstudentoutcomes andStaffhasopportunitiesto provideinputonhowtoimplement MTSS 3=Optimizing Item Instructional/intervention plansarenotdeveloped 24.Specific instructional/intervention plansaredevelopedand implementedbasedon verifiedreasonswhy studentsarenotmeeting academic,behaviorand social-emotional expectations andInstruction/Interventionplans consistentlyspecifywhatwillbe done,bywho,whenitwilloccur, andwherewithenoughdetailtobe 20 implemented andDataareusedtoverifythe reasonswhystudentsarenot meetingexpectations SAMVersion2.0(LastrevisedOctober2015)Page-7 Instructional/interventionplans aredeveloped Reasonswhystudentsare Reasonswhystudentsarenot notmeetingexpectationsare meetingexpectationsare notidentified identified andInstructional/intervention plansconsistentlyaredeveloped basedonverifiedreasonsstudents arenotmeetingexpectations andThereasonswhystudentsare notmeetingexpectationsspan multiplereasonsrelatedto instructionandthelearning environmentofwhystudents struggleandareverifiedusinga rangeofassessmentmethods andThegapbetweenexpectedand andThegapbetweenexpectedand currentoutcomesisidentified, currentoutcomesisidentified andisassociatedwithacademic, relativetoacademic,behaviorand behaviorandsocial-emotionalgoals social-emotionalgoalsandisused toidentifytheappropriatelevel (tier)ofinstruction/intervention Rating Thegapbetweenexpectedand currentoutcomesisidentified, 3=Optimizing Data-basedproblemsolvingoccurs Data-basedproblemsolvingoccurs across2ofthefollowing3: acrossallofthefollowing: • atleast3contentareas • acrossallcontentareas • atleast75%ofgradelevels • allgradelevels • atleasttwotiers • alltiers 2=Operationalizing Data-basedproblemsolving occursacross1ofthefollowing 4: • atleast2contentareas(e.g., reading,behavior,socialemotional) • atleast50%ofgradelevels • asingletier • onlyacademicoutcomesor onlybehaviorandsocialemotionaloutcomes 23.Academic,behaviorand social-emotionaldataare usedtoidentifyandverify 19 reasonswhy studentsare notmeetingexpectations Dataonacademic,behavior, andsocial-emotional outcomesmaybecollected, butdata-basedproblemsolvingdoesnotoccur across: • academic,behaviorand social-emotionalcontent areas • anygradelevels • anytier 22.Acrossalltiers,dataare Thegapbetweenexpected usedtoidentifythe andcurrentstudent differenceor“gap”between outcomesisnotidentified expectedandcurrentstudent outcomesrelativeto academic,behaviorand social-emotionalgoals 21.Integrateddata-based 17 problemsolving forstudent academic,behaviorand social-emotionaloutcomes occursacrosscontentareas, 18 gradelevels,andtiers 1=Emerging/Developing Self-AssessmentofMTSSImplementation(SAM) 0=NotImplementing 4.DataBasedProblemSolvingDomain(Items21-27) 23 24 27.Resourcesforandbarriers 21 totheimplementationof MTSSareaddressedthrough adata-basedproblemsolving process Schoolleadershipdiscusses resourcesforandbarriersto implementationofMTSSanddoes oneofthefollowing: • collectsdatatoassess implementationlevels • developsactionplanstoincrease implementation andPatternsofstudent performanceacrossdiversegroups areidentified 2=Operationalizing andInmostcasesdataare collectedtomonitorstudent progressandinterventionfidelity SAMVersion2.0(LastrevisedOctober2015)Page-8 Data-basedproblemsolving Schoolleadershipdiscusses ofresourcesforandbarriers resourcesforandbarriersto toimplementationofMTSS implementationofMTSS doesnotoccur Dataonstudentoutcomesare collected 1=Emerging/Developing Plansformonitoringprogress towardexpectedstudent outcomesaredeveloped Self-AssessmentofMTSSImplementation(SAM) 0=NotImplementing Progressmonitoringdoes notoccurandstudent progressisnotevaluated 26.Data-basedproblemPatternsofstudent solvinginformshowpatterns performanceacrossdiverse ofstudentperformance groupsarenotidentified acrossdiversegroups(e.g., racial/ethnic,cultural,socialeconomic,language proficiency,disabilitystatus) areaddressed Item 25.Studentprogressspecific toacademic,behaviorand social-emotionalgoals specifiedinintervention plansaremonitored Rating Schoolleadershipdiscusses resourcesforandbarriersto implementationofMTSSanddoes bothofthefollowing: • collectsdatatoassess implementationlevels • developsactionplanstoincrease implementation andDataonstudentoutcomes informshowMTSS implementationeffortsare impactingdifferentgroupsof students 3=Optimizing andChangesaremadeto instruction/interventionbasedon studentresponses Item 1=Emerging/Developing 2=Operationalizing Self-AssessmentofMTSSImplementation(SAM) 0=NotImplementing 3=Optimizing 30.Tier2(supplemental) Tier2strategiesarenot academicpractices existthat developedand/orclearly includestrategiesaddressing defined integratedcommonstudent needs,arelinkedtoTier1 27 instruction ,andare monitoredusing assessments/datasources tieddirectlytotheacademic, behaviorandsocialemotionalskillstaught 29.Tier1(core)behaviorand Tier1strategiesarenot social-emotionalpractices developedand/orclearly existthatclearlyidentify defined school-wideexpectations, social-emotionalskills instruction,classroom 25 managementpractices , andschool-widebehavior 26 andsocial-emotionaldata emotionalpractices.) 28.Tier1(core)academic Tier1elementsarenot practicesexistthatclearly developedand/orclearly 22 identifylearningstandards , defined 23 school-wideexpectations forinstructionthatengages students,andschool-wide 24 assessments • commonstudentneeds • linktoTier1instruction • linktobehaviorandsocialemotionalcontent/instruction • assessments/datasourceslink directlytotheskillstaught • commonstudentneeds • linktoTier1instruction • linktobehaviorandsocialemotionalcontent/instruction • assessments/datasourceslink directlytotheskillstaught SAMVersion2.0(LastrevisedOctober2015)Page-9 Tier2strategiesincorporate 2or3ofthefollowing4: • clearlydefinedschool-wide expectations • classroommanagementpractices • linktoTier1academic content/instruction • accessingschool-widebehavior andsocial-emotionaldata • clearlydefinedschool-wide expectations • classroommanagement practices • linktoTier1academic content/instruction • accessingschool-wide behaviorandsocial-emotional datasources Tier2strategiesincorporate 1ofthefollowing4: Tier1strategiesincorporate 2or3ofthefollowing4: • clearlydefinedlearning standards • school-wideexpectationsfor instructionandengagement • linktobehaviorandsocialemotionalcontent/instruction • assessments/datasources Tier1elementsincorporate 2or3ofthefollowing4: Tier1strategiesincorporate 1ofthefollowing4: • clearlydefinedlearning standards • school-wideexpectationsfor instructionandengagement • linktobehaviorandsocialemotional content/instruction • assessments/datasources Tier1elementsincorporate 1ofthefollowing4: Rating • commonstudentneeds, • linktoTier1instruction • linktobehaviorandsocialemotionalcontent/instruction • assessments/datasourceslink directlytotheskillstaught Tier2strategiesincorporate allofthefollowing: • clearlydefinedschool-wide expectations • classroommanagementpractices • linktoTier1academic content/instruction • accessingschool-widebehavior andsocial-emotionaldata Tier1strategiesincorporate allofthefollowing: • clearlydefinedlearning standards • school-wideexpectationsfor instructionandengagement • linktobehaviorandsocialemotionalcontent/instruction • assessments/datasources Tier1elementsincorporate allofthefollowing: 5.ThreeTieredInstructional/InterventionModelDomain(Items28-33)(Itemsinthissectionalternatebetweenaddressingacademic,behaviorandsocial- 25 26 33.Tier3(intensive)behavior Tier3strategiesarenot andsocial-emotional developedand/orclearly 30 practices includeintegrated defined strategiesthataredeveloped basedonstudents’needs andstrengths,arealigned withTier1andTier2 instructionalgoalsand strategies,andaremonitored usingassessments/data sourcesthatlinkdirectlyto skillstaught 32.Tier3(intensive)academic Tier3strategiesarenot 29 practices existthatinclude developedand/orclearly integratedstrategiesthatare defined developedbasedon students’needs,arealigned withTier1andTier2 instructionalgoalsand strategies,andaremonitored usingassessments/data sourcesthatlinkdirectlyto skillstaught 2=Operationalizing Tier2strategiesincorporate 2or3ofthefollowing4: SAMVersion2.0(LastrevisedOctober2015)Page-10 Tier3strategiesincorporate 2or3ofthefollowing4: • basedonstudents’needsacross academic,behaviorandsocialemotionaldomains • alignedwithTier1andTier2 instruction • linktoacademic content/instruction • assessments/datasourcesthat linkdirectlytotheskillstaught • developedbasedonstudents’ needsacrossacademic,behavior andsocial-emotionaldomains • alignedwithTier1andTier2 instruction • linktobehaviorandsocialemotionalcontent/instruction • assessments/datasourcesthat linkdirectlytotheskillstaught • developedbasedonstudents’ needsacrossacademic, behaviorandsocial-emotional domains • alignedwithTier1andTier2 instruction • linktobehaviorandsocialemotionalcontent/instruction • assessments/datasourcesthat linkdirectlytotheskillstaught Tier3strategiesincorporate 1ofthefollowing4: • basedonstudents’needs acrossacademic,behaviorand social-emotionaldomains • alignedwithTier1andTier2 instruction • linktoacademic content/instruction • assessments/datasourcesthat linkdirectlytotheskillstaught Tier3strategiesincorporate 2or3ofthefollowing4: Tier3strategiesincorporate 1ofthefollowing4: • commonstudentneeds • linktoTier1instruction • linktoacademic content/instruction • assessments/datasourceslink • assessments/datasourceslink directlytotheskillstaught directlytotheskillstaught • commonstudentneeds • linktoTier1instruction • linktoacademic content/instruction 1=Emerging/Developing Tier2strategiesincorporate 1ofthefollowing4: Self-AssessmentofMTSSImplementation(SAM) Item 0=NotImplementing 31.Tier2(supplemental) Tier2strategiesarenot behaviorandsocialdevelopedand/orclearly emotionalpracticesexistthat defined addressintegratedcommon studentneeds,arelinkedto 28 Tier1instruction ,andare monitoredusing assessments/datasources tieddirectlytotheskills taught commonstudentneeds linktoTier1instruction linktoacademiccontent assessments/datasourceslink directlytotheskillstaught Rating Tier3strategiesincorporate allofthefollowing: • basedonstudents’needsacross academic,behaviorandsocialemotionaldomains • alignedwithTier1andTier2 instruction • linktoacademiccontent/ instruction • assessments/datasourcesthat linkdirectlytotheskillstaught • developedbasedonstudents’ needsacrossacademic,behavior andsocial-emotionaldomains • alignedwithTier1andTier2 instruction • linkedtobehaviorandsocialemotionalcontent/instruction • monitoredusing assessments/datasourcesthat linkdirectlytotheskillstaught Tier3strategiesincorporate allofthefollowing: • • • • 3=Optimizing Tier2strategiesincorporate allofthefollowing: Item Nopoliciesandprocedures areinplace 35.Policiesandprocedures fordecision-makingare establishedforthe administrationof assessments,accessto existingdatasources,and 31 useofdata Theleadershipteamhaspolicies andproceduresfordecisionmakingthatincludeschedulesfor screening,useofdiagnostic assessments,progressmonitoring frequency,andcriteriafor determiningtier(s)ofsupport needed Stafflearnthepurposesof assessmentwithinMTSS andtheleadershipteamselects measuresforthepurposesof assessmentacrossacademic, behaviorandsocial-emotional areasthatarereliable,valid,and accessible,aswellasculturally, linguistically,and developmentallyappropriate andStaffconsistentlyadminister assessments,accessdatasources andmakedata-baseddecisions usingthepoliciesandprocedures fordecision-makingwithfidelity • determinestudentattainmentof academic,behaviorandsocialemotionaloutcomes andStaffengageinassessment withfidelityto: • answerpredetermined guiding/criticalquestions regardingstudent functioning/outcomes • identifystudentswhoareat-risk atleast3-4times/year • determinewhyastudentisatrisk • monitorstudent growth/progress • informinstructional/intervention planning 2=Operationalizing SAMVersion2.0(LastrevisedOctober2015)Page-11 Staffdonotunderstandand haveaccesstoacademic, behaviorandsocialemotionaldatasourcesor thataddressthepurposesof assessment 34.Staffunderstandandhave accesstoacademic,behavior andsocial-emotionaldata sourcesthataddressthe followingpurposesof assessment: • identifystudentsat-risk academically,socially, and/oremotionally • determinewhystudentis at-risk • monitorstudentacademic andsocial-emotional growth/progress • informacademicandsocialemotional instructional/intervention planning • determinestudent attainmentofacademic, behaviorandsocialemotionaloutcomes 1=Emerging/Developing Self-AssessmentofMTSSImplementation(SAM) 0=NotImplementing 6.Data-EvaluationDomain(Items34-39) Rating andAdherencetoandeffectiveness ofpoliciesandproceduresfor decision-makingareevaluated regularlyforefficiency,usefulness, andrelevanceforstudentsandstaff, anddataareusedtomake adjustmentstothepolicies andTheleadershipteamand/or staffcollaborativelyand systematicallyevaluateandrefine (asneeded)criticalguiding questionsandadjustassessment practicestoensureavailabilityof accurateandusefuldatatoinform instruction;assessmenttoolsare evaluatedforcontinuedvalue, usefulness,andcultural,linguistic, anddevelopmental appropriateness 3=Optimizing 27 28 andTheleadershipteam periodicallyconductsanalysesto determineconsistencyand accuracyofdata andProcessesandcriteriafor resourceallocationarerefined basedonstrategiesthatresultin improvedstudentoutcomes. andTheLeadershipteam periodicallyconductsanalysesto determinehowimplementationof criticalelementsofMTSSrelateto positivestudentoutcomes SAMVersion2.0(LastrevisedOctober2015)Page-12 Pleaseciteas: Stockslager,K.,Castillo,J.,Brundage,A.,Childs,K.,&Romer,N.(2016).Self-AssessmentofMTSS(SAM).Florida’sProblemSolving/Responseto InterventionProjectandFlorida’sPositiveBehaviorInterventionandSupportProject,UniversityofSouthFlorida andTheleadershipteamusesa protocol(e.g.emailnotificationsfor failuretotakeattendance,etc.)to monitordataconsistencyand accuracy Datasourcesarenot monitoredforaccuracyor consistency 39.Datasourcesare monitoredforconsistency andaccuracyincollection andentryprocedures Theleadershipteamensuresthat staffunderstandtheimportance ofaccurateandconsistentdata collectionpracticesandhave providedprofessional developmentonpoliciesand proceduresformethods,types andfrequencyofdatacollection Resourcesarenotallocated Resourcesareallocatedbasedon andtherelationshipbetweenthe basedonstudentneedand studentneed resourcesallocatedandthe theavailabilityoftime, outcomesofstudentsisevaluated availablepersonnel,funding, andmaterials 38.Availableresourcesare allocatedeffectively andTheleadershipteamusesdata sourcestoevaluateimplementation andtomakesystemic improvementstothecritical elementsofMTSS Theleadershipteamhas identifieddatasourcesthatwill beusedtoevaluate implementationofthecritical 6 elementsofMTSS 37.Datasources areusedto Nodatasourcestoevaluate evaluatetheimplementation implementationofthe andimpactofMTSS criticalelementsofMTSS havebeenidentified 31 Rating 1=Emerging/Developing 2=Operationalizing 3=Optimizing Theleadershipteamensures andStaffusethedatatoolsandare andDatatoolsareperiodically availabilityoftoolsthatcantrack providedassistanceasneeded assessedandthenecessarychanges andgraphicallydisplayacademic, aremadeinordertoimprove behaviorandsocial-emotional functionality,efficiency,and data,andstaffaretrainedonthe usefulness,andstaffisproficient useofthetoolsandontheir andindependentwithdatatools responsibilitiesfordata andeasilysupportnewstaff collection,entryand members management Self-AssessmentofMTSSImplementation(SAM) Item 0=NotImplementing 31 36.Effectivedatatools are Staffdonothaveaccessto usedappropriatelyand toolsthatefficientlyprovide independentlybystaff dataneededtoanswer problemsolvingquestions foracademic,behaviorand social-emotionalissues Self-AssessmentofMTSSImplementation(SAM)Endnotes 1. Instructional support staff may include: interventionists, coaches, behavioral specialists. Studentsupportpersonnelarecomprisedofschoolpsychologists,schoolcounselors,social workersandschoolnurses. 2. ResponsibilitiesforfacilitatingMTSSimplementationarenotlimitedto,butcaninclude: • • • • Promoting a school-wide vision and mission for MTSS implementation, including the developmentanddisseminationofaschool-wideimplementationplan Allocating resources (e.g., time, personnel, materials) for the planning and delivery of evidence-basedassessment,instructionandintervention Providingongoingprofessionaldevelopmentandcoachingsupporttoschoolstaff CollectingandanalyzingdataonMTSSimplementationefforts 3. Professionaldevelopmentandcoachingareongoingactivitiesthatdevelopthecapacityof staff to implement MTSS. Efforts should be aligned with results of school needs assessmentsandmodifiedbasedontheresultsofprofessionallearning. 4. AstrategicplanforMTSSimplementationshouldaddressthefollowingcomponents(ata minimum): • • • 5. Communicationandcollaborationstrategies Capacitybuildingtargetsandactivities DatatomonitorimplementationfidelityofthecriticalelementsofMTSS Different approaches to facilitating school-wide implementation of an MTSS model can include: • • The focus on a three-stage model of consensus building, infrastructure development, and implementationofpracticesconsistentwithanMTSSmodel The focus on a specific sets of activities related to successful implementation of a designated model of service delivery (e.g., National Implementation Research Network framework) The approach to facilitating school-wide implementation of an MTSS model should be connectedtotheSchoolImprovementPlan(SIP),aswellotherschool-wideplans.Ifyour district/state has provide guidance on an approach to implementing MTSS, then school leadershipteamsshouldconsiderusingthespecifiedapproach. 6. CriticalelementsofMTSScommunicatedtostaffinclude: • • • • 7. Curriculumstandards Assessmentdatausedtoinforminstruction Multipletiersofinstructionandintervention Data-basedproblem-solvingusedtomakedecisions “Coaching” is defined as technical assistance and support provided to school staff to improve implementation of components of an MTSS model (e.g., engaging in data-based problem solving, use of assessment data, development of multi-tiered instruction and intervention),including: • • Co-planning Modeling/demonstration • Co-facilitation • Guidedpracticewithhighqualityfeedback SAMVersion2.0(LastrevisedOctober2015)-Page1 29 8. Independentpracticewithguidedreflection • Instruction and intervention design and delivery that reflect student diversity make learning relevant and effective for all students by evaluating and accounting for diverse students’ culture, language,backgrounds,beliefs,knowledge,skillsandcontexts.Inaculturallyresponsiveschool effectiveteachingandlearningoccurinaculturally-supported,learner-centeredcontext,where studentstrengthsareidentifiedandutilizedtopromotestudentoutcomesacrossalltiers. 9. Schools will need to establish and communicate the problem solving process to be used, specificstepstobefollowed,andcriteriatousewhenmakingdecisions(e.g.,whatisgood, questionable,orpoorRtI?).Schoolsshouldconsiderdistrictandstateguidelineswhenavailable. 10. Processesandproceduresforproblemsolving,datacollectionanduse,anddecision-rules include: • Specificguidelinesonthestepsofproblem-solvingtobeused • Documentationrequirements • Opportunities for engaging in data-based problem-solving (e.g., Professional LearningCommunities,InterventionTeams) • Rolesandresponsibilitiesofparticipants 11. Resources encompass not only available monetary assets but also available personnel, instructionalmaterialsandtimethatwillfacilitatetheimplementationandsustainmentof anMTSSasaframeworkforsupportingallstudents. 12. Staff refers to employees at the school that will be impacted by or will be involved in implementation of MTSS. This will always include administration, teachers, other professional (e.g. student support services personnel) and para-professional support staff. The degree to which other employees (e.g. bus drivers, cafeteria workers, administrative supportstaff,facilitiesstaff)areincludedmaybedeterminedbytheirlevelofinvolvement with/implementationofMTSScomponentsattheindividualschoollevel. 13. Efforts to engage staff should align with district and state guidance regarding MTSS implementation to facilitate staff understanding of connections between school, district, andstateinitiatives. 14. Dataonstudentoutcomes,school-levelimplementationfidelity,thecapacityofeducators to implement, and commitment from staff are needed to inform implementation. Staff rolesandresponsibilitieswilldrivethespecificdatatheyneedtoinformimplementation. 15. Familyandcommunityengagementistheactiveandmeaningfulpartnershipthateducators buildandmaintainwithstudents’familiesandthebroadercommunityforthepurposeof supportingstudentlearning. 16. Intensiveoutreachtounresponsivefamiliesreferstoadditionalactivitiesundertakenbythe school to engage families of students who need additional supports, but who are not engaging with the school's typical outreach practices (e.g. letters and phone calls home). Intensive outreach is an individualized approach requiring information gathering and problemsolvingtoidentifyoutreachstrategiesthataremorelikelytobesuccessfulforafamily. SAMVersion2.0(LastrevisedOctober2015)-Page2 30 17. Data-based problem solving refers to a multi-step process that includes examining performance related to goals/expectations (Problem Identification), understanding variables causing problems (Problem Analysis), selecting/designing and implementing strategies to lessen barriers and achieve goals (Instruction/Intervention Delivery), and monitoringeffectiveness(Monitoring/Evaluation) 18. Data-basedproblem-solvingshouldoccur(a)acrosscontentareas(reading,math,science, behavior, social-emotional and other relevant content areas for a school) (b) within and acrossgradelevels(e.g.,horizontalmeetingsfor6th,7th,8th,aswellasverticalmeetings for6ththrough8th),and(c)acrosstiers(performancedatainresponsetoinstructionusedto engage in problem-solving for all students [Tier 1], for some students receiving supplementalinstruction[Tier2],andforstudentsreceivingindividualizedsupport[Tier3]). 19. Reasons why students are not meeting expectations are sometimes referred to as hypothesesorbarrierstolearning.Thebigideaisthatschoolsidentifypotentialcurriculum, instruction, environmental (e.g., peer distractions, classroom management issues), and learner (e.g., skill deficits) for why the student is not meeting expectations and collect data/informationtodeterminewhichreasonsarecontributingtotheproblem. 20. Specificinstruction/interventionplansincludeinformationoutlining: • • • • • • • • Thegoaloftheintervention/actionplan What intervention or action steps (e.g., curriculum adjustments, instructional processes andprocedures)willbeputinplace Howoften(daily/weekly/etc.)theinterventionwillbeutilized Howlongeachsessionistobeimplemented Whoisresponsibleforinterventionimplementationandsupport Whereandwhentheinterventionwillhappen Planformonitoringinstruction/interventionfidelityandprogresstowardsidentifiedgoals Timeframe(dates)forperiodicreviewofprogressmonitoringdataanddecisionpoints 21. Structured problem solving is utilized to identify resources that can be used to facilitate implementation and barriers that are hindering implementation for the purpose of developingspecificactionplanstoincreaseimplementationlevels. 22. Prioritylearningstandardsarecurriculumstandardsthatdefinewhatstudentsshouldknow and be able to do for a given content area and grade level (e.g., Common Core State Standards;statespecificstandards,Social-EmotionalLearningStandards). 23. Expectationsforinstructionoftenincludeelementsrelatedtotheinstructionalroutine(e.g., whole-group, small-group, and independent practice), amount of time dedicated to instruction,andwhichevidence-basedinstructionalstrategiesareused. 24. Both statewide assessments and formative assessments administered to all students are important to identify so that expectations for the data needed to inform decisions are consistent. SAMVersion2.0(LastrevisedOctober2015)-Page3 31 25. Structuredinstructionofbehavioralexpectationsandsocialandemotionalskillsisprovided to all students. Classroom routines include social and emotional learning principles and classroom management strategies embedded into instruction. School climate and environments support student well-being. A small number of clearly defined school-wide expectations that are positively stated are a foundational element of Tier 1 school-wide behaviorsupportsystem. 26. School-wide social-emotional behavior data may include Office Discipline Referrals, InSchool Suspensions, Out-of-School Suspensions, and social-emotional screening data sourcesusedtoexaminetheeffectivenessofTier1behaviorandsocial-emotionalsupports. 27. Tier 2 interventions should be aligned with Tier 1 instructional goals and expectations, address high-probability barriers to achieving instructional goals and expectations, and includeassessmentswhichmeasurespecificskills,generaloutcomes,andstudentprogress. 28. Tier 2 interventions should be aligned with school-wide behavior and social-emotional expectations, address high-probability barriers to meeting instructional goals and student well-being, and include assessments that monitor student discipline incidents, socialemotionalskills,andwell-being. 29. Tier 3 interventions generally provide increased exposure (time in minutes) to quality instructionorintervention,morefocusedinstructionmatchedtostudentneed,andsmaller groupings.Additionally,Tier3interventionsoftenaredevelopedduringindividualstudent focused problem-solving sessions. Importantly, Tier 3 interventions focused on academic issuesshouldbelinkedtoTier1and2instructionalcontentandprocessesandalsoshould considerwhatbehavioralorsocial-emotionalsupportsareneededforsuccess. 30. Tier 3 interventions are matched to a student’s specific behavior and social-emotional needsandensurethestudenthasaccesstoTier1andTier2supports.Forafewstudents withcomplexneeds,individualizedinterventionsmayinvolvewraparoundsupportsacross systems(e.g.,mentalhealth,education,medical,family,etc.).Individualizedinterventions include specific prevention and consequence-based strategies based on assessment information (i.e., Functional Behavior Assessment), and may include modifications to the classroom environment or instruction, teaching new skills, and reinforcement of desired behaviorsaswellasarangeofsupportssuchasmentalhealthservices. 31. District and states typically create or adopt data management systems. They also specify access and use requirements. School leaders should coordinate with district and state leaders to understand requirements and establish and communicate procedures for using the data system at their school. Quality indicators for data management systems include: real-timerelevantdataforacademic,behavior,andsocial-emotionalcontent,theabilityto graphically represent data, provision of tiered intervention data, integrated academic, behavior,andsocial-emotionaldata,andthedataarecustomizableattheschoollevel. SAMVersion2.0(LastrevisedOctober2015)-Page4 32 SAMScoringSheet THISSCORINGSHEETHASABBREVIATEDLANGUAGEANDSHOULDONLYBEUSEDTORECORDANDDISPLAYRESPONSES Rateeachitemonascalefrom0-3(0=NotStarted;1=Emerging/Developing;2=Operationalizing;3=Optimizing) 1.LeadershipDomain(Items1-5) Rating0-3 1.Theprincipalisactivelyinvolved 0 1 2 3 2.Aleadershipteamisestablished 0 1 2 3 3.Theleadershipteamactivelyengagesinongoingprofessionaldevelopment 0 1 2 3 4.AstrategicplanforMTSSimplementationisdeveloped 0 1 2 3 5.Theleadershipteamisactivelyfacilitatingimplementation 0 1 2 3 2.BuildingtheCapacity/InfrastructureforImplementationDomain(Items6-16) Rating0-3 6.ThecriticalelementsofMTSSaredefinedandunderstood 0 1 2 3 7.Professionaldevelopmentandcoachingprovidedtostaff 0 1 2 3 8.TheleadershipteamfacilitatesPDondata-basedproblem-solving 0 1 2 3 9.TheleadershipteamfacilitatesPDonmulti-tieredinstructionandintervention 0 1 2 3 10.CoachingisusedtosupportMTSSimplementation 0 1 2 3 11.Schedulesprovideadequatetimefortrainingandcoaching 0 1 2 3 12.Schedulesprovideadequatetimetoadministerassessments 0 1 2 3 13.Schedulesprovideadequatetimeformultipletiersofinstruction/interventions 0 1 2 3 14.Schedulesprovideadequatetimefordata-basedproblem-solving 0 1 2 3 15.Processes,procedures,anddecision-rulesareestablishedforDBPS 0 1 2 3 16.ResourcestosupportMTSSimplementationareidentifiedandallocated 0 1 2 3 3.CommunicationandCollaborationDomain(Items17-20) Rating0-3 17.StaffhaveconsensusandengageinMTSSImplementation 0 1 2 3 18.StaffareprovideddataonMTSSfidelityandstudentoutcomes 0 1 2 3 19.Theinfrastructureexiststosupportfamilyandcommunityengagement 0 1 2 3 20.EducatorsactivelyengagefamiliesinMTSS 0 1 2 3 4.Data-BasedProblem-SolvingDomain(Items21-27) Rating0-3 21.DBPSforstudentoutcomesoccursacrosscontentareas,gradelevels,andtiers 0 1 2 3 22.Acrosstiers,datausedtoidentify“gap”betweenexpectedandcurrentoutcomes 0 1 2 3 23.Dataareusedtoidentifyreasonswhystudentsarenotmeetingexpectations 0 1 2 3 24.Plansbasedonverifiedreasonswhystudentsarenotmeetingexpectations 0 1 2 3 25.Studentprogressspecifictoacademicorbehaviorgoalsaremonitored 0 1 2 3 26.Dataareusedtoaddressperformanceacrossdiversegroup 0 1 2 3 27.ResourcesforimplementationofMTSSareaddressedthroughdata-basedproblem-solving 0 1 2 3 5.ThreeTieredInstructional/InterventionModelDomain(Items28-33) Rating0-3 28.Tier1academicpracticesclearlyidentifylearningstandards 0 1 2 3 29.Tier1behaviorpracticesidentifyschool-wideexpectations 0 1 2 3 30.Tier2academicpracticesincludecommonstudentneeds,arelinkedtoTier1 0 1 2 3 31.Tier2behaviorpracticesincludecommonstudentneeds,arelinkedtoTier1 0 1 2 3 32.Tier3academicpracticesarebasedonstudents’needs,alignedwithTier1andTier2 0 1 2 3 33.Tier3behaviorpracticesarebasedonstudents’needs,alignedwithTier1andTier2 0 1 2 3 6.Data-EvaluationsDomain(Items34-39) Rating0-3 34.Staffunderstandandhaveaccesstodatasources 0 1 2 3 35.Policiesandproceduresfordecision-makingareestablished 0 1 2 3 36.Effectivedatatoolsareusedappropriatelyandindependentlybystaff 0 1 2 3 37.Datasourcesareusedtoevaluatethefidelityandimpact 0 1 2 3 38.Availableresourcesareallocatedeffectively 0 1 2 3 39.Datasourcesaremonitoredforconsistencyandaccuracy 0 1 2 3 SAMVersion2.0(LastrevisedOctober2015) 33 ActionPlanandGuidingQuestions 1. InwhichdomainsarethegreatestgapsincurrentandoptimalMTSSimplementation? 2. WhichspecificitemsrepresentthegreatestgapsincurrentandoptimalMTSS implementation? 3. WhichspecificMTSSimplementationactionsoractivitieswillyourteamfocuson improvingwithinyourschool? 4. Whicharemostimmediatelyactionable? 5. Whichwouldbemostimpactful? 6. Whichwouldbemostfoundational(alignedwithwhereyouwanttobe)? Youmaychoosetouseyourownactionplanningformortheoneprovidedbelow. Action/Activity Whois responsible? Whenwillit bestarted? Whenwillitbe completed? When/howwill weevaluateit? SAMVersion2.0(LastrevisedOctober2015) 34 Appendix B. Additional Considerations for SAM Collection, Analysis, and Interpretation This section provides additional information relative to SAM use and interpretation. Specifically, SBLT team composition and time commitment guidelines (prerequisites for completing the SAM), administration considerations, suggestions for identifying strengths and needs, and additional examples of district-level data analysis are provided. Prerequisites School-Based Leadership Team (SBLT). SBLTs are comprised of approximately six to eight staff members selected to take a leadership role in facilitating MTSS implementation in a school. Staff included on the SBLT should have the following roles represented: administration, general education teachers, special education teachers, student services, and content specialists (e.g., reading, math, behavior). SBLT members should receive training on the MTSS model including strategies for facilitating implementation. Individuals on the team also should adopt roles and responsibilities to ensure efficient and productive planning and problem-solving meetings. Important responsibilities include a facilitator, timekeeper, data coach, and note-taker, in addition to providing expertise in the particular content areas or disciplines listed above. Time and commitment. Prior to committing to completion of the SAM, SBLT members should consider the time and resources necessary to collect, analyze, and action plan around the SAM results. Results from a national pilot study indicated that average completion time for the SAM averaged between 1-2 hours. Administration Data collection and action planning. During the SAM administration, SBLT members should consider prioritizing completion of the full SAM instrument prior to engaging in action planning. Completion of the entire set of 39 items allows teams to review their full implementation data and prioritize action planning based on having all of the information available. Frequency of use and administration time of year. As described earlier, it is recommended that SBLT members complete the SAM at least once per year. Completion of the SAM annually allows team members to evaluate initial MTSS implementation efforts, as well as changes in implementation levels across years. Collecting the SAM once per year also allows SBLT members to re-prioritize their focus based on results of the data collection. SBLTs also should consider the time and resources dedicated to increasing MTSS implementation when deciding on the frequency of SAM data collection. For example, if an SBLT completes the SAM at the beginning of the school year and dedicates a significant amount of resources (e.g., professional development, coaching, etc.) toward increasing specific MTSS implementation components, an end of year SAM administration might help the SBLT measure their progress and re-organize their MTSS focus for the upcoming school year. Interpreting SAM data and applying the results Identifying strengths and needs. • • • The extent to which schools should be implementing MTSS practices will depend on training received; length of time since the school decided to implement the model; district, state, and national policies and procedures; availability of data systems to support data-based decision-making; among myriad other factors. Given the multiple interacting variables that impact school efforts to implement any initiative, it is important to consider all aspects of the system that contribute to or impede engagement in specific activities while developing plans that address needs evident from the data. In addition to the SAM, SBLT members should consider additional MTSS implementation data and information that are available prior to action planning. For example, schools may already collect behavior implementation data (e.g., Benchmarks of Quality [BOQ]) that can be used in conjunction with the SAM. SBLT members should consider the SAM to be one piece of data in examining a complex system of educational service delivery. SBLTs should consider the identification of areas of strength, areas of potential strength, and areas for improvement. For areas for improvement, schools should identify those components that are foundational (i.e., must be in place for other components to be present) to facilitate optimal levels of MTSS implementation. Additional district-level example graphs Below are sample district-level graphs to provide an example of how aggregate, district-level SAM results can be displayed for easier interpretation. The examples are based on real SAM data and decisions made by the District Leadership Team in the district that completed the SAM. 35 Figure 4. District-Level Example of SAM Data: Average Domain Scores. Explanation of the Graph Sunshine School District recently committed to implementing MTSS district-wide. The newly formed district leadership team (DLT) at Sunshine School District met at the beginning of the school year to plan for implementation, but realized that they did not know where to begin. At the suggestion of the district’s MTSS lead, the team decided to have all the schools complete the SAM at their next SBLT meeting to inform both district and school goals and activities for the year. After all of the schools completed the SAM, the team first reviewed the district average domain scores to identify potential areas of strengths and need (Figure 4). A quick review of the graph led the team to identify communication/collaboration, data/evaluation and capacity/infrastructure as potential areas to focus on related to improvement of MTSS implementation. The team decided to look at the averages across levels (elementary, middle, and high) to provide further information to assist in selecting a focus area. 36 Figure 5. District-Level Example of SAM Data: Aggregated Elementary Domain Scores. Explanation of the Graph Upon examination of the aggregate scores across all elementary schools, the DLT at Sunshine School District noted the same domains (communication/collaboration, data/evaluation and capacity/infrastructure) as potential areas to focus on related to improvement of MTSS implementation. Figure 6. District-Level Example of SAM Data: Aggregated Middle School Domain Scores. 37 Explanation of the Graph The data aggregated across all middle schools indicated similar relative areas of need (communication/collaboration and data/evaluation) as the overall district average. The DLT noted that all aggregate middle school domains (except Three-Tier model) were below the district domain average of 1.5 and, therefore, were potential targets for improvement. Figure 7. District-Level Example of SAM Data: Aggregated High School Domain Scores. Explanation of the Graph Similar to the district and elementary data, the aggregate domain scores across all high schools indicated the same potential areas for improvement (communication/collaboration, data/evaluation and capacity/infrastructure). Interpretation and use of the data While the DLT noted similar levels of implementation across many of the domains, the team decided to prioritize building the capacity and infrastructure for implementation due to it being foundational for many of the other activities and/or practices within the other domains to occur. In order to better understand the broad domain, the team next looked at the distribution of responses for each item within the Building Capacity/Infrastructure domain. 38 Figure 8. District-Level Example of SAM Data: Item Frequencies (Building Capacity/Infrastructure Domain). Explanation of the Graph As mentioned above, the Sunshine District DLT decided to focus on the Building Capacity/Infrastructure domain after a review of their SAM scores. Figure 8 above includes the district-wide distribution of results for the SAM Building Capacity/Infrastructure items. Interpretation and use of the data Examining items within a SAM domain. Following the first administration of the SAM at the beginning of the year, the DLT met to discuss the results and plan for addressing building capacity and infrastructure for implementation. The team examined the district-wide distribution of ratings across each item (Figure 8). Within the Building Capacity/Infrastructure domain, the DLT found the lowest percentage of schools rated themselves as optimizing and the highest percentage of schools rated themselves as emerging/developing on item 8 (Professional development and coaching for staff members on data based problem solving relative to their job roles/responsibilities), item 9 (Professional development and coaching for all staff on multi-tiered instruction and intervention relative to their job roles/responsibilities) and item 15 (Processes, procedures, and decision-rules are established for data-based problem solving). Identifying specific needs. Overall, these data indicated that across schools there was a lack of clear expectations about the circumstances under which educators are to engage in data-based problem solving to address student and systems issues. Additionally, the data indicated that a lack of explicit professional development focused on how to engage in data-based problem solving (e.g., process, data sources) and how to provide multi-tiered supports to meet students needs existed. A strategic district-wide focus to build capacity in these areas is foundational for sustainable MTSS and is a logical first step for the Sunshine District. The DLT planned for addressing these issues and agreed to examine progress following the next SAM administration. Once district-wide expectations and capacity have been established for common areas of need, the district will consider how to best address school level-specific needs, such as overall low levels of implementation across all domains at middle schools. 39 Appendix C. Standardized Factor Loadings for Items on the SAM Table 4. Standardized Factor Loadings and Standard Errors for Items on the SAM. Factor Item # Item Estimate Leadership 1 The principal is actively involved in and facilitates MTSS implementation A leadership team is established that includes 6-8 members with cross-disciplinary representation (e.g., principal, general and special education teachers, content area experts, instructional support staff, student support personnel) and is responsible for facilitating MTSS implementation The leadership team actively engages staff in ongoing professional development and coaching necessary to support MTSS implementation A strategic plan for MTSS implementation is developed and aligned with the school improvement plan The leadership team is actively facilitating implementation of MTSS as part of their school improvement planning process The critical elements of MTSS are defined and understood by school staff The leadership team facilitates professional development and coaching for all staff members on assessments and data sources used to inform decisions The leadership team facilitates professional development and coaching for staff members on data-based problem solving relative to their job roles/responsibilities The leadership team facilitates professional development and coaching for all staff on multitiered instruction and intervention relative to their job roles/responsibilities Coaching is used to support MTSS implementation Schedules provide adequate time for trainings and coaching support Schedules provide adequate time to administer academic, behavior and social-emotional assessments needed to make data-based decisions Schedules provide adequate time for multiple tiers of evidence-based instruction and intervention to occur Schedules provide adequate time for staff to engage in collaborative, data-based problem solving and decision making Processes, procedures, and decision-rules are established for data-based problem solving Resources available to support MTSS implementation are identified and allocated Staff have consensus and engage in MTSS Implementation .69 Standard Error .04 .80 .03 .71 .03 .82 .02 .93 .01 .76 .03 .77 .02 .79 .02 .83 .02 .75 .68 .03 .03 .68 .03 .74 .03 .68 .03 .85 .02 .83 .02 .84 .02 2 3 4 5 Capacity/ Infrastructure 6 7 8 9 10 11 12 13 14 15 16 Communication and Collaboration 40 17 18 19 Data-Based Problem Solving 20 21 22 23 24 25 26 27 Three-Tiered Model 28 29 30 31 32 Staff are provided data on MTSS implementation fidelity and student outcomes The infrastructure exists to support the school’s goals for family and community engagement in MTSS Educators actively engage families in MTSS Integrated data-based problem solving for student academic, behavior and social-emotional outcomes occurs across content areas, grade levels, and tiers Across all tiers, data are used to identify the difference or “gap” between expected and current student outcomes relative to academic, behavior and social- emotional goals Academic, behavior and social-emotional data are used to identify and verify reasons why students are not meeting expectations Specific instructional/intervention plans are developed and implemented based on verified reasons why students are not meeting academic, behavior and social-emotional expectations Student progress specific to academic, behavior and social-emotional goals specified in intervention plans are monitored Data-based problem solving informs how patterns of student performance across diverse groups (e.g., racial/ethnic, cultural, social-economic, language proficiency, disability status) are addressed Resources for and barriers to the implementation of MTSS are addressed through a data-based problem solving process Tier 1 (core) academic practices exist that clearly identify learning standards, school-wide expectations for instruction that engages students, and school-wide assessments Tier 1 (core) behavior and social-emotional practices exist that clearly identify school-wide expectations, social-emotional skills instruction, classroom management practices, and school-wide behavior and social-emotional data Tier 2 (supplemental) academic practices exist that include strategies addressing integrated common student needs, are linked to Tier 1 instruction, and are monitored using assessments/data sources tied directly to the academic, behavior and socialemotional skills taught Tier 2 (supplemental) behavior and socialemotional practices exist that address integrated common student needs, are linked to Tier 1 instruction, and are monitored using assessments/data sources tied directly to the skills taught Tier 3 (intensive) academic practices exist that include integrated strategies that are developed based on students’ needs, are aligned with Tier 1 and Tier 2 instructional goals and strategies, and are .80 .03 .70 .03 .66 .83 .03 .02 .85 .02 .83 .02 .82 .02 .84 .02 .62 .03 .81 .02 .79 .03 .81 .03 .90 .02 .91 .01 .88 .02 41 33 Data and Evaluation 34 35 36 37 38 39 42 monitored using assessments/data sources that link directly to skills taught Tier 3 (intensive) behavior and social-emotional practices include integrated strategies that are developed based on students’ needs and strengths, are aligned with Tier 1 and Tier 2 instructional goals and strategies, and are monitored using assessments/data sources that link directly to skills taught Staff understand and have access to academic, behavior and social-emotional data sources that address the following purposes of assessment: • identify students at-risk academically, socially, and/or emotionally • determine why student is at-risk • monitor student academic and social-emotional growth/ progress • inform academic and social-emotional instructional/intervention planning determine student attainment of academic, behavior and social-emotional outcomes Policies and procedures for decision-making are established for the administration of assessments, access to existing data sources, and use of data Effective data tools are used appropriately and independently by staff Data sources are used to evaluate the implementation and impact of MTSS Available resources are allocated effectively Data sources are monitored for consistency and accuracy in collection and entry procedures .86 .02 .87 .02 .84 .02 .79 .02 .84 .02 .80 .82 .02 .02
© Copyright 2025 Paperzz