DAC Meeting January 20, 2011 3:30 – 7:30 Presenters: Mark Smith, Bill Horwath, Dina Hasiotis, Carla Stevens, Ben Hernandez, Robin Licato Attendees: Drew Bissell, Boon Chew, Jose Contreras, Joe Espinoza, Reba Goodman, Mary Hacopian, Jewel Jefferson, Earl Jimmison, Christine Johnson, Linder Johnson, Keith Kalman, Danette Maldonado, Matilda Orozco, Laurie Parkin, Martha Salazar‐Zamora, Rachele Vincent, [In attendance but did not sign‐
in: Rhonda Jones, Nimfa Gallardo, Ruth Henderson] Meeting Notes: 1. Meeting Introduction: Mark Smith 2. Discussion on Accepting the Academic Calendar‐ Mark Smith 3. ETI Work‐ Introduction and Welcome: Bill Horwath ƒ Bill Horwath talks, gives introduction about meeting. ƒ Dina Hasiotis: [Overview of process timeline, overview of upcoming public comment period] 4. Presentation on Proposed Measures of Student Learning: Carla Stevens – HISD lead for MSL WG. [Explains membership.] Ben Hernandez (ES teacher). [Explains earlier work of MSL WG. Examined 3 different school districts that use SP in teacher evaluation. Discussed strengths, limitations, different measures (EVAAS, student work product), looked at teacher placements and best‐fit and second‐best‐fit measurements.] Robin Licato [Explains – not just a 3‐teacher representation. Went back to campuses, talked to other teachers – although 3 physical bodies of teachers, we are trying to represent ‘teacher’.] Summary of 11 types of measures in constructing a performance measure based on SL. – see slide. Primary aim was to use individual measures of teacher progress, as opposed to group measures. Discusses different type of individual growth measures. Explanation of strengths and limitations of absolute vs. value‐added growth on TAKS assessments along with principal‐approved assessments. Overview of high‐level assumptions that guided work group, specifically related to value‐added measures: e.g.ranking of teachers by EVAAS – explains ASPIRE awards, effect of EVAAS – vs. in evaluation, you don’t want to rank‐order teachers by some sort of threshold for purposes of contract‐
renewal, for example – looking instead at, say, standard deviations for purposes of determining effective vs. developing – what makes sense to determine performance pay may not make sense to determine appraisals – even though data/score is the same – difference is how value is placed on score. Question: A teacher could do well on evaluation but still end up with $0 on ASPIRE award, right? Response: Potentially, yes, you would want to keep that, for example, average teachers or above‐
average vs. rewarding excellence for a incentive program. It should be looked at differently because the impact is different, decisions based on that data is different for the 2 different systems. How we put EVAAS into appraisal system should be different than how we put it into ASPIRE. [Overview of assurances re: use of value‐added in performance appraisals] Question: So using EVAAS data as part of their rating, why do we have to put teachers on a curve – what if all teachers are getting 2 years of growth, so some will be low‐performing. Carla Stevens: There are different ways to calculate and are investigating with SAS EVAAS. When talking about teachers’ contracts, different purpose. Still want to use district’s reference gain to look at. Question: Why don’t we just look at student gain? Carla Stevens: We have many measures, only one is value‐added, so we do have student again. EVAAS is only 1 of 6. And everyone will have a minimum of 2. So that is part of what we’re looking at, what would be the best measure. We are not saying some people will be looked at only with EVAAS. [Overview of recommended measures for teacher appraisal – crossing types of teachers with measures available. Referred to handout of PowerPoint.] Question: What are we looking at/comparing teachers to? Response: This is not comparing campuses to each other. With district‐wide assessments, we haven’t settled on what we’re comparing against. Example of IB schools – is it other IB schools, other Houston schools? Comment: The reason I bring up is because in a dual‐language school – they have to be compared against teachers who are doing the same thing – dual vs. native English vs. native Spanish will all look different. Presenter: You look at what data is available to determine how to measure. Comment: So I pick one or the other? Response: No, you use both – you have to use 2 at a minimum. You don’t want to use 4 or 5, it’s too much, but the guidance we got from SDMCs and DAC is that we don’t want anyone getting eval on just 1 measure of SL. [Explanation of the tables for that assign measures to grades and subjects based upon the assessments available] Comment: Going back to the first measure – sufficient stretch. … As a principal – how much discretion will I have? I don’t think it’s fair in a dual‐language academy to compare teachers against others who aren’t doing the same thing. Response: You are using teacher as control – compared against self. All teachers have ability to be above‐average – unlike what we use for ASPIRE award. Presenter: In magnet vs. non‐magnet, some classrooms have large level of students with commended score from prior year – so I wanted to discuss stretch of TAKS achievement test. In issues of classrooms with a large percentage of high‐performing students, there would be an attainment measure, as long as they maintain high‐achievement – for example, the chance of getting kids at 96 to 98, it is very difficult – there are other industries where maintenance is acceptable and where that says you’re effective. Presenter: So there are different times when different ways of using EVAAS makes sense. Maintaining high rates of commended vs. looking at average value‐added score. Teacher Working Group Member: (MSL Working Group member): One district we looked at had a circle graph where the graph changed as data changed – we don’t know what the pie looks like, but in our head we get to a place where should MSL piece be as large when you’re in subjects like Sculpting when there is no real reliable way to data that as compared to person who’s accountable to 5th grade, you have to do this TAKS to progress to 6th grade, so if you’re a teacher who teaches in a tested subject – that’s your job as a teacher. You have to be in a position to be held accountable. If we’re going to say, teachers have to do pre‐ and post‐test, then we can compare – so, German 1, who’s going to pass that?! So, we just keep going back to – the pie’s going to change. What could you collect and test? Even as data becomes more reliable – would a 1st year teacher in a tested subject – should their gains be as large as a 5th year teacher? 1st year scores should have more wiggle room vs. a 6th year. By then, I should be more accountable – or, if teachers switch subjects, the pie has to change Carla Stevens: In reviewing the guiding principles you set up for us : {Carla reviews} o Guidelines: Did we do what you wanted us to do? o Multiple measures – yes o Include student work products – Yes o Use growth‐based measures – YEs o Allow appraiser to consider extenuating circumstances – Yes o SDMCs: Ensure measures are equally rigorous for all teachers – ƒ The Working Group struggled with this. It’s subjects/grades where we don’t have test scores. There’s no way of getting around subjectivity, except with lots of training, calibration, norming. But we’re still going to have a hard time with that. Carla Stevens asks the group: Did we hear you? Are we moving in the right direction? DAC Response: Yes. Question: Go through the last one again [ensure measures are equally rigorous]. Response: – if you have your professionalism score, your Instructional Practice score – if one teacher has a great Student Performance score – that piece might count more – vs. another teacher who has more subjective measures where you can’t control for error in it as much, so you rely on other scores to round out the evaluation more than you would with another teacher. We’re trying to manage the tension between everyone having the same weight against what’s most fair and accurate. So, rather than everyone having the same weights, it’s potentially better to go in the direction of fairness and accuracy and trusting measures you’re most confident in for that particular type of teacher. Dina Hasiotis: To take a step back, part of the SDMCs’ and DAC’s responsibility is to develop criteria. So the Working Group took guidance—The DAC should look for what is fair in the conditions for each teacher type of teacher in this, and decide what a recommendation should be moving forward. Around the weights question, that’s something for the cabinet to decide, taking into consideration all these pieces around fairness, accuracy. But, besides the weights, what would be the most fair measures of teachers in these categories is what we need to come to consensus on. We would like to hear questions that you all have. This has been a lot, so we want to open up to questions. Question: We looked at a lot of possibilities for teachers who have different subjects, etc. Is this going to be something where each school has a computer based sample? What are the mechanics of having an appraisal form? What I’m looking at here is a monster! Response: This isn’t the form – here’s the big a‐ha moment. Looking at slides – this is how we measure SL. All these check‐marks, how do we measure how much student has changed, grown, been impacted by the teacher – none of these are titled by teacher, we’re talking about kids. “If you are a child who … does this, takes Stanford, etc. – then, your teacher of record can get these pulled out of system to go into the Student Performance portion of evaluation. That’s separate from the evaluation of the appraiser walking into the room. Question: How do those numbers get to the assistant principal? Response: That’s the process working group, like Bill Horwath. A lot of that infrastructure is already in place. So if it helps folks, we’re going to have to build a technology solution around the new appraisal system – that’s in the works. One way to think about it – you have PDAS and AYP data – SL data will feed into the new system. Question: How are we going to use Stanford based data if we don’t take it until the end of the year in terms of rating? Response: They will all be from the year behind. Response: Another option being considered, is when you go through your appraisal, there are 3 components. At the end of the year, you can easily get your Instructional Practice and Professional Expectations ratings and, depending, you can also get a summary based on the measures you use – now, a lot of measures may not get in until the end of school year. Let’s say for 2011‐2012 – your 2011‐2012 rating may come to you in Aug, Sep – that final piece. Your rating will be, was based on last year. Response: Another way is to keep 3‐year running average. Question: What were the challenges you all found in trying to maintain equal rigor? Teacher Working Group Member: My wife is a music teacher. She is concerned because there is no quantifiable data for her, so she’s asking, “Is it going to be a student work project?” Her concern is that her principal may not understand music enough to objectively determine her students’ performance. So, one challenge was in cases where there is no data. Comment: There have to be tutorials. Response: We don’t want to create an assessment that is the same for every single subject. Comment: I’m a physical education teacher, I was assessed by someone from the health/PE department, then that was given to my principal. Response: We tried to cover everyone and discussed having people in those content areas and have those discussions with them, regarding what is something that we can consider? Our expertise is very limited, but you would fall under an appraiser‐approved assessment category. Question: Is there someone from the union on your Working Group? Response: Yes. Question: For my school, when the SDMCs submitted Student Learning criteria, none that were based on EVAAS or test scores – there was observations, are kids making connections, etc. So, I’m surprised that none of that appeared here. Response: Actual observations of student learning and seeing what students are doing in the classroom is imbedded within the Instructional Practice criteria. Comment: I’m thinking about current PDAS – Response: From our understanding, you have to have some numerical data that demonstrates students are learning. Do kids understand what is being instructed, that’s IP, what can be observed sitting in the room. The IP folks are taking that as student‐centered observations – so not what teachers are doing but what students are doing in the room. All three teachers on the Working Group are on SDMCs as well, so they had good perspective on their campuses’ SDMCs. Dina Hasiotis: Are you ready to put this out for public comment? That’s the next question that we would like you all to answer for your recommendation. The working group has figured out what could be student performance criteria based upon the SDMC recommendations. The question is, do you all feel this is good to move forward with? If anyone can stay after, we want also to discuss, what is a good way to communicate this out, but if you all feel this could serve as basis for Student Learning measures in the district – then it’s going to go back through the SDMCs to say, “we agree” or “we don’t agree” – but this is a recommendation on what could be included for Student Learning. That’s what we want to get a sense of from you all tonight. What other proof points could we share, is there something missing? Teacher Working Group Member: I have been thinking if this is totally against what my SDMC wanted? Dina Hasiotis: There’s a good point around value‐added and EVAAS – only a handful brought up “No value‐added” or “No EVAAS” in Cycle 2 – so is the current list of measures fair? Teacher Working Group Member: I think the problem is what the SDMCs are working with and think of when they think of EVAAS which is through ASPIRE which isn’t fair– here we’re using the power of EVAAS but we’re changing the criterion for what it spits out for appraising teacher vs. rewarding teachers (ASPIRE). Teacher Working Group Member : You mentioned your campus did not like the idea of using EVAAS. I don’t think some campuses liked it, but everybody wanted to use growth – the best way to measure growth is through value‐added because it takes into account more than just growth. So, it’s how district used or interpreted data. Comment: I think this is a great tool, a great document. It’s evident that you have researched it, that you have had conversations with other educators. That’s evident because you are teachers and you know what you want people to look at, how you want to be appraised. I vote, [laughter] I think it needs to be presented. But I will say, you are going to have to explain it as if I was in the 3rd grade so I can make sure everything is understood by everyone. Teachers are going to – they are concerned – not just something that has been forced on us – but everything needs to be explained clearly. Question to the DAC: Does anyone disagree? Comment from DAC member: Let’s move forward. Overall consensus: Yes to moving forward with the measures. Question: I think we should move forward but it is hard to grasp all of these ideas. Is there a way to simplify it? Comment: I think y’all are going to have a lot of teachers balking at this. If you confuse them up front, they’re going to shut down. Comment: I think the current explanation is wordy, but we can have summations. I think you can condense the document into a series of summations and make the full document available to lend credibility and transparency. There will be skeptics who say, oh no, you just gave me a watered down version. Question: The concern I hear from teachers is that they don’t know how value‐added data is retrieved, how do you get that data. They feel very concerned about that because they feel like, how do they know it’s accurate? They keep talking about how in other jobs not in education, when you’re given data like that you know how data is retrieved. Response: We have done a lot of training, but I can’t tell you that everybody gets it. We will continue to train on how value added training is calculated. We are also going to talk to SAS about including more data in the teacher reports to increase transparent. One idea is to have a roster of kids in the report, so that way you can look at those numbers and see how, directionally, that led to your rating. Comment: The problem is more transparency about how the data was actually compiled, how you derived the data. Not the rosters, not the previous scores, not the present scores, but how the data was actually calculated. Comment: How you convert it to NCES. Response: There’s no way we can show you a formula that you can calculate, because it’s not done that way. But we can try to give results and data to show pictures to show how it is fair across different groups of teacher. Just because it’s a complicated formula doesn’t mean it’s wrong. Complicated because it takes into consideration all the things you’ve asked us to take into consideration. Multiple test scores per child. Missing test data. All the variance of test scores. Which makes it something you can’t just show a little formula and here, go calculate it. If you want that out of this model, then go back to the one where it says TAKS attainment – you’re there. And look at all of the problems you have with it. If you want something that has more of these things controlled for, it will be a paper of statistical compilations that has more benefits, but still has some limitations. Yes, very difficult to understand methodology, but this takes into consideration everything TAKS absolute attainment doesn’t. Question from DAC member: Which classes have sufficient stretch and which ones don’t? Response: That is being determined with analysis. Comment from DAC member: When sharing this, do one presentation for the teachers, one for the parents. 5. Process and Synthesis Detailed Recommendations: Bill Horwath Bill Horwath: The Process and Synthesis Working Group, has been looking at the process you outlined and carefully thinking about what additional details could be added. We are going to review that tonight and have you finalize your Cycle 3 Recommendations on the process. [Detailed Process Handout reviewed in groups and discussed] Whole‐ Group Discussion: Comment: Second big bullet – 2 conferences – to have all of this information ready in time for all of my teachers– I will die on the job, as a principal. Response: Appraisers would do this, not just principals and as an appraiser I have to make sure all the administrators do their walkthroughs. Comment: I might only have one day to observe them, and then they might be absent. Response: There are really 5 opportunities: BOY, MOY, EOY, which are about setting goals. Then there’s also long observations each semester preceded by or followed by another conference. SDMCs asked for more feedback, and monthly observations. Comment: There’s a concern about the time it takes to do all these conferences, covering classes for teachers while they’re in conferences. If we can get rid of that 5 day requirement – some are happy to talk to me in the hall. If it’s in stone, and I mess up and fail to do the post‐conference, their evaluation is null and void. What about lengthening the time for conferences and feedback? Comment: Maybe take out the MOY conference? Response: Reminder that the MOY is where you’re going back to what you’re doing in classroom and reviewing goals, professional expectations feedback, professional development, etc, versus a post conference that is just about the classroom observation. Comment: Are those goals set for every individual student, which in a primary grade they might be – can goals be set by the algebra 1 team that we will have 90% of our students pass at the end of the fall semester, and 92% pass at end of spring semester. Are those acceptable goals? If I’m looking at passing data, we set goal of 90% and we’re at 88%, so we’re ambitious and yet let’s keep moving. So I’m not sure how goals get set. Comment: There’s flexibility on the part of the appraiser, to help teachers make the right targets for student learning. With a new teacher, you think about – there’s flexibility among appraisers. The joint effort to think what’s going to help track progress throughout the year. Something that’s organic on campus. Comment: In the elementary case where you only have 18 kids – vs. a HS math teacher – the way you set good goals may look different. Comment: Also, the transient nature of kids in a secondary setting. Bill Horwath: To summarize section 1 – 5 day requirement is too tight, getting the materials to folks ahead of the conference is tight. Those are restrictions that could be an issue in the evaluation. Comment: My concern is the timeframe – what happens if I miss something? Bill Horwath: What other areas of the draft did you discuss or would like to comment on? Comment: If you have 2 assessors from the same campus, there’s a chance you’re not getting objective feedback. Some teachers would be concerned because it could be subjective. There should be an opportunity to have somebody who has a background in what you’re teaching and they could come in and give you some objective feedback. Comment: Someone who does know your content helps feed your appraisal? The idea of a 2nd appraiser is a good one, but we’re not sure how it counts. How will it feed into final rating? Comment: – We need to clarify what is meant by “logs” – should say communication logs Comment: In types and frequencies of feedback, I heard from teachers they want 1 observation to be announced so they have chance to put their best foot forward. Comment: I would feel really bad if I announce it and that turns I couldn’t make it. Comment: If we are able to get additional observation on first page or how about teachers can request Comment: Realistically, I am not going to invite you, because I am so involved. The idea of one being announced, it is kind of a good thing, I think. Comment: A professional teacher is a professional teacher, all the time, regardless of whether or not they’re being observed. It shouldn’t be announced Overall Consensus: Leave as currently written, and if someone wants to request, they can, and feedback can be gathered from public comment. Walkthroughs with written feedback – Comment: On walkthroughs with written feedback, just say “within 10 days”. Comment: For the walkthrough component, would there be a standardized form used throughout district? Response: The forms piece is still to be considered. There are lots of opinions. Bill Horwath: With these changes, we will revise and send to you to ensure it reflects the discussion, and ask all DAC members to send any suggestions or changes prior to finalizing the recommendation. Comment: With the new DAC members coming, how will they understand the process we have gone through and that we can’t start from scratch? That is important, since we have done so much already. 6. Conclusion of the meeting Next meeting will be on February 17. Thank you.