Dallas ISD
Reasoning Mind
Results
2012–2013
Solving for every variable
Dallas ISD Reasoning Mind
Results 2012–2013
Summary
The Dallas Independent School District (Dallas ISD) implemented Reasoning Mind
district-wide in second and third grades during the 2012–2013 school year. The
implementation was non-standard, in that not every teacher received the recommended
support offered by Reasoning Mind.
IMPACT OF REASONING MIND
While we cannot isolate the impact of Reasoning Mind because we did not have a
control group in Dallas ISD, we found that, when holding previous performance fixed,
students who studied more Reasoning Mind objectives achieved higher third-grade
State of Texas Assessments of Academic Readiness (STAAR) scores and secondgrade Iowa Test of Basic Skills (ITBS) scores in mathematics. This pattern held across
all student ethnicities.
PROGRAM USE
Students averaged slightly over an hour per week of program use over the course of
the year (compared to a target of 2 hours).
TEACHER SUPPORT
While students whose teachers received Reasoning Mind support tended to be
weaker on the pretest, they saw greater increases than students whose teachers did
not receive support.
TWO-YEAR IMPACT OF REASONING MIND
We saw that the relationship between STAAR and ITBS scores and Reasoning Mind
objectives studied was sustained over a two-year period. The students who studied
more cumulative objectives during second and third grade tended to have higher
STAAR scores at the end of third grade.
TIME SPENT USING REASONING MIND OUTSIDE OF SCHOOL
Students spent 5% of their time using the Reasoning Mind system during
non-school hours.
TEACHER/ADMINISTRATOR OPINIONS
Both teachers and administrators saw value in the Reasoning Mind program. The
majority (more than 60%) stated they would like to see the program return to their
campuses.
1
CONTENTS
SUMMARY
1
BACKGROUND
3
Professional Development
4
Program Use
4
Problems Solved
5
STAAR SCALE SCORES
6
All Students
6
Impact by Student Groups
9
Full-Year vs. Partial-Year Reasoning Mind Students
12
“Supported” vs. “Unsupported” Students
12
Two-Year STAAR Results
14
Two-Year STAAR Results by Student Group
15
ITBS STANDARD SCORES
All Students
17
17
Impact by Student Group
20
Full-Year vs. Partial-Year Reasoning Mind Students
22
“Supported” vs. “Unsupported” Students
23
TIME SPENT ONLINE OUTSIDE OF REGULAR SCHOOL HOURS
25
SURVEY RESULTS
26
Value of Reasoning Mind
26
Impact on Students
27
2
Dallas ISD Reasoning Mind
Results 2012–2013
Background
The Dallas Independent School District
(Dallas ISD) began piloting Reasoning
Mind during the 2010–2011 academic
year. In 2011–2012, following a
successful pilot study, the program was
expanded to include all second graders
in Dallas ISD, and in the following
school year, all third graders began
using the program as well. Table 1
shows the number of students who
used the program, broken down by
grade level.
Table 1. Number of students in Dallas ISD
who used Reasoning Mind in 2012–2013.
Number of students
Grade
2
12,868
3
12,233
Total
25,101
Every student in these grades who logged
onto the system at least once by June 1,
2013, is included.
Reasoning Mind recommends that all teachers receive regular support from program
coordinators (PCs), especially in the first three years of teaching with the program. This
was the case in Dallas ISD during the 2011–2012 school year, but in 2012–2013, Dallas ISD
chose to have the vast majority of teachers (81%) use the program unsupported. Below is
the break-down of teacher support in Dallas ISD.
Table 2. Support level and total number of PC visits per teacher.
Support level
Number of teachers
PC visits scheduled,
total
PC visits carried out,
total
Light support
146
438
438
No support
629
0
0
Total
775
438
438
3
PROFESSIONAL
DEVELOPMENT
Reasoning Mind requires a 16-hour pre-service training for all teachers using the
program—the Qualification Course (QC). Out of 775 teachers who taught with Reasoning
Mind in Dallas ISD in 2012–2013, only 143 teachers (18%) completed the course before the
school year began. A further 146 teachers completed their training by October 4, 2012. By
January 18, 2013, an additional 238 teachers completed their pre-service training. The
remaining teachers completed the QC between January 19 and the end of the school year,
with the exception of 26 teachers who did not complete the QC at all.
A total of 144 teachers (18.6%) took part in Reasoning Mind’s professional development
workshops, for a total of 830 credits and an average of 5.76 credits per teacher.
PROGRAM USE
The scheduled time for Reasoning Mind
overall (including transitions) is 2 hours per
week. An established district
implementation of Reasoning Mind should
show 85% of scheduled time being spent
online, or 100 minutes per week. By May 1,
students in both second and third grades
in Dallas ISD spent roughly 65 minutes per
week online, corresponding to a deficit of
35%. This deficit was likely due to the late
launch of the program in the district
(launch dates varied between August 28,
2012, and May 20, 2013). In May, there was
a surge in usage, such that students
studied in Reasoning Mind for 120 minutes
per week online. This was nearly double
the rate of the first eight months of the
year.
4
Considering time online in hours, both second and third graders finished the year with 35
hours online (up to June 1) and 28 hours prior to May 1, on average. Even taking into
account the hours by June 1, this fell well short of the target of 70 hours. However, if the
level of performance during the month of May (an average of 7 hours online for the
month) were to be sustained for the entire school year, the average time online would
actually exceed 70 hours. The maximum time online achieved was 139 hours for a second
grader and 165 hours for a third grader.
Table 3. Time online for Dallas ISD students, broken down by grade, in hours, from
the beginning of the school year until May 1, 2013.
Grade
N
Mean
SD
Minimum
Maximum
2
12,868
28
14
0.01
139
3
12,233
27
14
0.01
165
Overall
25,101
—
—
—
—
PROBLEMS SOLVED
Students in second grade in Dallas ISD solved 568 basic problems by May 1, 2013, while
those in third grade solved 636. The record for the most problems solved in second grade
belongs to a student who solved 3,311 problems. The student with the most problems
solved in third grade solved an astonishing 3,626. Altogether, students in Dallas ISD
solved over 15 million problems in Reasoning Mind, for an average of 600. We note that
standard deviations for the number of problems solved are quite large, most likely due to
the large range of launch dates and subsequent large variation in hours online.
Table 4. The number of problems solved by students in Dallas ISD by May 1, 2013. Overall,
Dallas ISD students solved over 15 million problems in Reasoning Mind.
Grade
N
Mean
SD
Minimum
Maximum
2
12,868
568
323
1
3,311
3
12,233
636
404
1
3,626
Overall
25,101
601
366
1
3,626
5
Dallas ISD Reasoning Mind
Results 2012–2013
STAAR Scale Scores
Scores for the mathematics section of the State of Texas Assessments of Academic
Readiness (STAAR) were made available by the district for third graders of Dallas ISD in
2013. The STAAR is a criterion-referenced, state-mandated standardized exam aligned to
the Texas Essential Knowledge and Skills. To eliminate students who may have transferred
outside the district, we include only students who were given at least a single problem by
June 1, 2013. There were 12,233 such students in third grade. We were able to match
STAAR scale scores and Reasoning Mind metrics for 10,822 of the students. Below is the
analysis of their results.
ALL STUDENTS
The table on the next page shows the
satisfactory and advanced rates on the
math STAAR for Dallas ISD third graders
assigned to Reasoning Mind in 2013 as
well as program use metrics for
third-grade students. As the table
shows, students solved 655 problems by
May 1—roughly 94 problems for each
month of instruction until then. During
the month of May, students solved 189
problems on average (see Table 5).
There was also a noticeable difference in the time spent online prior to May 1 and after
May 1. In the month of May, students studied an average of three Reasoning Mind
objectives and spent an average of 8 hours online. In comparison, the monthly average
prior to May (7 months, October–April) was 1.6 Reasoning Mind objectives and 4 hours
online. That is, in the month of May the use of the Reasoning Mind program doubled.
6
While we are pleased that students were able to learn more math within the Reasoning
Mind system, because this increase happened after state standardized tests, this increase
in usage cannot contribute to student performance on the STAAR.
Table 5. Math STAAR results and internal Reasoning Mind metrics for all third-grade
Reasoning Mind students for whom test scores were available.
Variable
% passing, 2013 math STAAR
% “advanced,” 2013 math STAAR
Total A-level1 problems by May 1, 2013
Mean
SD
Minimum
Maximum
N
58.1
10,822
9.2
10,822
655
402
0
3,626
10,822
% A-level problems correct by May 1, 2013
72
13
0
100
10,822
Hours online by May 1, 2013
28
14
0
165
10,822
11
7
0
44
10,822
844
485
5
4,126
10,822
% A-level problems correct by June 1, 2013
74
13
15
98
10,822
Hours online by June 1, 2013
36
17
0.1
198
10,822
Reasoning Mind objectives studied
by June 1, 2013
14
9
0
50
10,822
Reasoning Mind objectives studied
by May 1, 2013
Total A-level problems solved by June 1, 2013
Reasoning Mind objectives studied
To get a graphical look at the relationship between 2013 STAAR scale scores and
Reasoning Mind objectives studied throughout the year, while still taking into account
previous performance, we graphed this relationship for several different groups of
students with similar previous performance. That is, we grouped students who all had
similar Iowa Tests of Basic Skills (ITBS) standard scores in second grade (e.g., students
who scored between 120 and 140 on the ITBS). The ITBS is a nationally standardized,
norm-referenced test that measures and compares the achievement of students in
subjects such as mathematics.We then graphed the relationship between 2013 STAAR
scale scores and Reasoning Mind objectives studied for each of these groups. To keep the
groups of Reasoning Mind objectives studied similar with regard to the number of
students in each group, we split Reasoning Mind objectives studied into quartiles
according to the Reasoning Mind objectives actually studied this year, rather than
grouping students with regard to the total Reasoning Mind objectives available. As an
example, the 25% of students who studied the fewest Reasoning Mind objectives would
be in the 1st quartile. The results are in Chart 1. The dotted lines are the Level II passing
and Level III advanced cut offs.
1
A-level is the basic Reasoning Mind problem level.
7
When graphing the relationship in this way, we are able to look at groups of students
broken down by previous performance. We found that for each of the groups, students
who studied more Reasoning Mind objectives saw higher average STAAR scale scores.
The increases from the 1st to the 4th quartiles of Reasoning Mind objectives are especially
pronounced for the students who came in with the least prior preparedness (ITBS
standard score 120–140) and those with the strongest prior preparedness (ITBS standard
score 201–220). This performance pattern may be due to a variety of factors. One such
factor may be the ability of the Reasoning Mind program to differentially instruct
students. It allows the students who need the most help to receive more scaffolding and
the students who need enrichment to work on more difficult problems. Further analyses
will be needed to confirm this.
Chart 1.
1800
2012 ITBS
Standard Score
1703
1700
2013 STAAR Scores
1631
1600
201–220
1668
Advanced
1603
1538
1544
1563
181–200
161–180
1513
141–160
1500
1422
1400
1438
1449
120–140
1398
Passing
1343
1350
1321
1300
1291
1325
1283
1200
1231
0-5
1241
6-10
11-15
16 or more
Reasoning Mind objectives studied (quartiles)
When running a regression to take previous test scores into account, we found a
significant relationship between Reasoning Mind objectives studied and student
performance. This indicates that the more Reasoning Mind objectives a student studies,
the better the student does on the STAAR (see Table 6).
8
Reasoning Mind objectives studied is often a better indicator of use of the Reasoning
Mind system than hours online because it is a measure of the material covered in the
Reasoning Mind system rather than how much time a student is in the system. Put simply,
the greater the number of mathematical topics learned in the Reasoning Mind system, the
better a student tends to do.
Table 6. Regression table predicting STAAR scale scores from Reasoning Mind objectives
studied and second-grade ITBS standard scores.
Reasoning Mind objectives
studied before May 1, 2013
Second-grade ITBS standard
scores
R2
B
SE
ß
t
.54
2.36
0.132
.122
17.87*
6.11
0.061
.690
100.89*
* p < .05
IMPACT BY
STUDENT GROUPS
One question that is always a concern
when there is a diverse student
population is whether a program benefits
all (as opposed to just some) groups of
students. To address this question, we did
two analyses. First, we graphed the mean
difference in STAAR scale scores between
the quartile of students with the most
Reasoning Mind objectives studied and
the quartile of students with the least
Reasoning Mind objectives studied by
2012 ITBS standard score and student
ethnicity (see Chart 2). We found that in
all of the ethnic groups, students in the
top quartile of Reasoning Mind objectives
studied outperformed students in the
bottom quartile by a substantial margin.
This was especially pronounced for
African American students, particularly
those who had the highest and lowest
2012 ITBS standard scores.
9
Difference in STAAR scores between students
with 16 or more objective studied and students
with 0-5 Reasoning Mind objectives studied
Chart 2.
160
140
120
100
80
60
40
20
0
-20
-40
2012 ITBS
score between
121-140
Asian
2012 ITBS
score between
141-160
African
American
2012 ITBS
score between
161-180
Hispanic/
Latino
2012 ITBS
score between
181-200
White
2012 ITBS
score between
201-220
In addition to the graph above, we also report all the mean 2013 STAAR scale scores by
2012 ITBS standard score and student ethnicity (see Table 7). The results of this analysis
are similar to those above.
10
Table 7. Impact of Reasoning Mind by student ethnicity.
Ethnicity
Asian
Reasoning Mind
objectives
studied
121-140
141-160
161-180
181-200
201-220
0-5
—
1354
1374
—
—
6-10
—
1303
1452
1566
—
11-15
1163
1377
1497
1535
—
—
1356
1504
1615
1750
0-5
1239
1266
1365
1455
1418
6-10
1238
1297
1384
1495
1600
11-15
1374
1313
1412
1496
1595
16 or more
1375
1370
1425
1515
1546
0-5
1225
1301
1409
1521
1627
6-10
1223
1329
1428
1530
1613
11-15
1261
1348
1443
1550
1652
16 or more
1264
1346
1452
1568
1698
0-5
1184
1278
1426
1595
1705
6-10
—
1343
1432
1611
1718
11-15
—
1342
1440
1569
1739
1288
1365
1474
1572
1758
16 or more
African
American
Hispanic/
Latino
White
2012 ITBS standard score between
16 or more
11
FULL-YEAR VS. PARTIAL-YEAR
REASONING MIND STUDENTS
Of 11,520 students for whom 2013 math STAAR data were available, 3,816 (33%) were
full-year Reasoning Mind students. We defined full-year students as those in a classroom
that began using Reasoning Mind by October 1, 2012 who spent at least 10 hours using
the program by May 1, 2013. We chose May 1 as the cutoff because the STAAR exams were
administered in mid-to-late April and material covered after May 1 would not be helpful
on the exam. We note that these students have higher metrics, both on the STAAR and
Reasoning Mind. On the STAAR, they outperformed their partial-year counterparts by 1.5
percentage points (see Table 8).
Table 8. Math STAAR results and internal Reasoning Mind metrics for full-year and
partial-year third-grade Reasoning Mind students.
Variable
Full-year
Partial-year
59.1
57.6
9.5
9.0
764
601
Mean % A-level problems correct by May 1, 2013
73
72
Mean hours online by May 1, 2013
32
26
Mean Reasoning Mind objectives studied by May 1, 2013
12
10
% passing, 2013 math STAAR
% “advanced,” 2013 math STAAR
Mean A-level problems solved total by May 1, 2013
“SUPPORTED” VS. “UNSUPPORTED”
STUDENTS
In the fall of 2012, Dallas ISD switched to a model in which only one teacher on each
campus was eligible for Reasoning Mind support, due to the district’s budget constraints.
Principals chose which teacher this would be. Intuitively, it would make sense for the
classrooms with the greatest need to be chosen for support. We have evidence that
confirms that principals directed support to the teachers of the least prepared students:
on the pretest, the “supported” students had an ITBS passing rate that was 4 percentage
points lower than that of the “unsupported” students. By 2013, the gap narrowed to 3
percentage points.
12
Table 9. Third-grade cohort standardized test passing rates and internal metrics, broken
down by teacher support level.
No support ( N =1950)
Mean
SD
Half-support ( N =338)
Mean
SD
Support (N =1472)
Mean
% passing,
2013 math STAAR
62
62
59
% passing,
2012 math ITBS
65
63
61
Total A-level
problems by May 2013
SD
769
418
563
253
811
423
% A-level problems
correct by May 2013
74
12
72
14
74
12
Hours online by
May 2013
32
15
23
8
36
12
Reasoning Mind
objectives studied
by May 2013
13
8
9
5
13
7
967
488
737
327
994
498
% A-level problems
correct by June 2013
76
12
74
14
76
12
Hours online by
June 2013
39
17
30
10
43
15
Reasoning Mind
objectives studied
by June 2013
17
9
12
7
16
9
Total A-level
problems by June 2013
13
TWO-YEAR STAAR RESULTS
Third-grade Dallas ISD students are the first cohort in Dallas ISD to use Reasoning Mind
for two years. This allows us to look at the cumulative influence of Reasoning Mind over
the course of those two years. To do this, we graphed student 2013 STAAR scale scores by
several different groups of students with similar previous performance. That is, we
grouped students who all had similar 2011 ITBS standard scores in first grade (e.g.,
students who scored between 120 and 140 on the 2011 ITBS). We then graphed the
relationship between 2013 STAAR scale scores and cumulative Reasoning Mind objectives
studied (Reasoning Mind objectives studied by June of 2012 plus Reasoning Mind
objectives studied by May of 2013) for each of these groups. To keep the groups of
cumulative Reasoning Mind objectives studied similar with regard to the number of
students in each group, we split the cumulative Reasoning Mind objectives studied into
quartiles according to the Reasoning Mind objectives actually studied between 2011 and
2013, rather than grouping students with regard to the total Reasoning Mind objectives
available. As an example, the 25% of students who studied the fewest cumulative
Reasoning Mind objectives would be in the 1st quartile. The results are in Chart 3. The
dotted lines are the Level II passing and Level III advanced cut offs.
When graphing the relationship in this
way, we are able to look at groups of
students broken down by previous
performance. We found that for each of
the groups, students who studied more
Reasoning Mind objectives over the two
years they used Reasoning Mind had
higher average STAAR scale scores (see
Chart 3).
Every group of students with similar 2011 ITBS standard scores trended towards higher
2013 STAAR scale scores when more Reasoning Mind objectives were studied. This was
especially true of the highest performing students on the 2011 ITBS (those with a score
between 181 and 200). The students in the group with between 22 and 29 Reasoning Mind
objectives studied over the two years (about two-thirds of the Reasoning Mind objectives
for one year) actually had 2013 STAAR scale scores slightly lower than their peers in the
next level of prior achievement. However, the quartile of students who studied the highest
number of Reasoning Mind objectives greatly outperformed all the other students.
14
Chart 3.
1800
2011 ITBS score
1723
181-200
1700
161-180
Mean 2013 STAAR Scores
Advanced
1600
1570
1521
1500
1492
1587
141-160
1557
1515
121-140
1471
1443
1411
1400
1377
1371
Passing
1348
1321
1300
1283
1200
0-21
22-29
30-38
39 or more
Objectives studied (quartiles)
TWO-YEAR STAAR RESULTS
BY STUDENT GROUP
To make sure Reasoning Mind is benefitting all student ethnic groups, we did two
analyses. First, we graphed the mean difference in 2013 STAAR scale scores between the
quartile of students with the most Reasoning Mind objectives studied and the quartile of
students with the least Reasoning Mind objectives studied by 2011 ITBS standard score
and student ethnicity (see Chart 4). We found that in all of the ethnic groups, students in
the top quartile of Reasoning Mind objectives studied outperformed students in the
bottom quartile by a substantial margin. This was especially pronounced for African
American students, particularly those who had the highest 2011 ITBS standard scores.
15
Difference in STAAR scores
between students with 39 or more
objectives studied and students
with 0-21 objectives studied
Chart 4.
160
2011 ITBS
score between
121-140
140
120
100
2011 ITBS
score between
141-160
80
60
40
2011 ITBS
score between
161-180
20
0
African
American
Hispanic/
Latino
White
In addition to the graph above, we also report all the mean 2013 STAAR scale scores by
2011 ITBS standard score and student ethnicity (see Table 10). The results of this analysis
are similar to those above.
Table 10. Impact of Reasoning Mind by student ethnicity.
Ethnicity
African
American
Hispanic/
Latino
White
Reasoning Mind
objectives
studied
2011 ITBS standard score between
121-140
141-160
161-180
0-5
1251
1319
1400
6-10
1301
1378
1473
11-15
1316
1405
1477
16 or more
1345
1441
1533
0-5
1295
1397
1520
6-10
1324
1418
1529
11-15
1351
1448
1565
16 or more
1373
1474
1585
0-5
1422
1638
6-10
1447
1564
11-15
1484
1639
1514
1652
16 or more
16
Dallas ISD Reasoning Mind
Results 2012–2013
ITBS Standard Scores
ALL STUDENTS
We performed cohort analysis for all second-grade Reasoning Mind students in Dallas ISD
who were assigned to Reasoning Mind for whom two years’ worth of ITBS standard scores
were available—first grade in 2012 and second grade in 2013, scores with computation.2 As
the table below shows, student grade equivalents grew, on average, by roughly one year
on the sections excluding computation and one year and one month on the section
including computation.
Table 11. ITBS results for the entire second-grade Reasoning Mind cohort.
2012
2012
2013
2013
Mean
SD
Mean
SD
149
13
168
17
Math ITBS NCE without computation
48
20
49
21
Math ITBS grade equivalent without
computation
1.2
0.69
2.3
1.00
150
11
169
15
49
21
50
22
2
0.61
3
0.92
Variable
Math ITBS standard score without
computation
Math ITBS standard score with computation
Math ITBS NCE with computation
Math ITBS grade equivalent with
computation
Students who are tested on grade level are given computation questions, while those
who are tested at the kindergarten level in first grade are not and hence do not have
scores that include computation. These students were excluded from analysis. Here we
do not present students who took modified versions of the ITBS, since those groups had
fewer than 1% of students each.
2
17
Reasoning Mind objectives studied
To get a graphical look at the relationship between 2013 ITBS standard scores and
Reasoning Mind objectives studied throughout the year, while still taking into account
previous performance, we graphed the difference between students’ 2013 ITBS grade
equivalent scores and their 2012 ITBS grade equivalent scores by Reasoning Mind
objectives studied. To keep the groups of Reasoning Mind objectives studied similar with
regard to the number of students in each group, we split Reasoning Mind objectives
studied into quartiles according to the Reasoning Mind objectives students actually
studied this year, rather than grouping students with regard to the total number of
Reasoning Mind objectives available. As an example, the 25% of students who studied the
fewest Reasoning Mind objectives would be in the 1st quartile. The results are in Chart 5.
From this graph, we can see that for every jump in Reasoning Mind objectives studied,
there was a corresponding jump in grade equivalent. In fact, the 25% of students who
studied the most Reasoning Mind objectives saw an average increase of about 1.4 grade
equivalents.
Mean difference in ITBS Grade Equivalent
Chart 5.
1.4
1.6
1.4
1.2
1.0
0.9
1.0
1.2
0.8
0.6
0.4
0.2
0.0
0-12 objectives
13-17 objectives
18-22 objectives
23 or more objectives
Reasoning Mind objectives studied (quartiles)
Because looking simply at student growth can be misleading (for example, higher
performing students sometimes grow more than lower performing students, and at the
same time it is likely that higher performing students are able to study the most
Reasoning Mind objectives), we graphed this relationship in another way to verify that all
groups of students were growing at similar rates. We grouped students who had similar
ITBS standard scores in second grade (e.g., students who scored between 120 and 140 on
the ITBS). We then graphed the relationship between 2013 ITBS scale scores and
Reasoning Mind objectives studied for each of these groups. To keep the groups of
Reasoning Mind objectives studied similar with regard to the number of students in each
group, we split Reasoning Mind objectives studied into quartiles according to
18
the Reasoning Mind objectives actually studied this year, rather than grouping students
with regard to the total Reasoning Mind objectives available. As an example, the 25% of
students who studied the fewest Reasoning Mind objectives would be in the 1st quartile.
The results are in Chart 6.
When graphing the relationship in this way, we are able to look at groups of students
broken down by prior performance. We found that for all levels of previous performance,
the greater the number of objectives studied the higher the test score. That is, students
that completed the most objectives (more than 22) had higher test scores that those who
completed fewer objectives.
Chart 6.
200
193
194
195
2013 ITBS Scores
180
182
2012 ITBS score
between 161-180
184
175
2012 ITBS score
between 141-160
162
2012 ITBS score
between 120-140
170
170
167
164
158
160
150
2012 ITBS score
between 181-200
189
190
180
197
155
149
140
0-12
13-17
18-22
23 or more
Reasoning Mind objectives studied (quartiles)
When running a regression to take previous test scores into account, we found a
significant relationship between Reasoning Mind objectives studied and student
performance. This indicates that the more Reasoning Mind objectives a student studies,
the better the student does on the ITBS (see Table 12).
Reasoning Mind objectives studied is often a better indicator of use of the Reasoning
Mind system because it is a measure of the material covered in the Reasoning Mind
system rather than simply how much time a student is in the system. Put simply, the
greater the number of mathematical topics learned in the Reasoning Mind system, the
better a student does.
19
Table 12. Regression table predicting 2013 ITBS standard scores from Reasoning Mind
objectives studied and first-grade ITBS standard scores.
Reasoning Mind objectives
studied before May 1, 2013
R2
B
SE
ß
t
.562
0.027
0.001
.228
34.45*
0.963
0.010
.639
96.43*
First-grade ITBS standard
scores
* p < .05
IMPACT BY
STUDENT GROUP
To address if Reasoning Mind had the same impact on sub-populations of students, we
graphed the change in ITBS grade equivalents from 2012 to 2013 by Reasoning Mind
objectives studied for each student ethnicity.3 We found that for every student group,
students with more Reasoning Mind objectives studied saw a greater average increase in
ITBS outcomes (see Chart 7).
Mean Difference in ITBS Grade
Equivalent from 2012 to 2013
Chart 7.
1.8
1.6
1.4
1.2
1.0
0.8
0.6
0.4
0.2
0.0
0-12 objectives
13-17 objectives
18-22 objectives
23 or more objectives
Reasoning Mind objectives studied (quartiles)
Asian
African American
Hispanic/Latino
White
We removed American Indian, Pacific Islander, two or more, and no response students
because each of these groups had fewer than 5 participants in one or more categories.
3
20
In addition to the graph above, we also report all the mean 2013 STAAR scale scores by
2012 ITBS standard score and student ethnicity (see Table 13). The results of this analysis
are similar to those above.
Table 13. Impact of Reasoning Mind by student ethnicity.
Ethnicity
Asian
African
American
Hispanic/
Latino
White
Reasoning Mind
objectives
studied
2012 ITBS standard score between
121-140
141-160
161-180
181-200
0-12
147
165
179
—
13-17
150
167
183
213
18-22
156
171
188
—
23 or more
149
174
198
—
0-12
148
161
175
198
13-17
154
166
178
185
18-22
158
168
182
181
23 or more
160
173
186
188
0-12
150
165
181
179
13-17
155
166
182
197
18-22
158
170
183
191
23 or more
162
174
188
196
0-12
154
166
186
199
13-17
154
172
191
204
18-22
160
173
191
200
23 or more
165
178
195
202
21
FULL-YEAR VS. PARTIAL-YEAR
REASONING MIND STUDENTS
Out of 11,534 students for whom 2013 math ITBS data were available, 2,561 (22.2%) were
full-year Reasoning Mind students. We defined full-year students as students in a
classroom that began using Reasoning Mind by October 1, 2012, and who spent at least 10
hours using the program by May 1, 2013. We chose May 1 as the cut-off because the ITBS
exams are given in April and material covered after May 1 would not be helpful on the
exam. On the ITBS, they outperformed their partial-year counterparts by 5–6 points
(depending on if one includes performance on the computation section in the score; Table
14).
Table 14. 2013 math ITBS results and internal Reasoning Mind metrics for full-year and
partial-year second-grade Reasoning Mind students.
Variable
Full-year
Partial-year
Math ITBS standard score without computation
167
162
Math ITBS NCE without computation
49
42
3
2
169
163
50
42
3
3
701
118
Mean % A-level problems correct by May 1, 2013
74
77
Mean hours online by May 1, 2013
20
6
9
4
Math ITBS grade equivalent without computation
Math ITBS standard score with computation
Math ITBS NCE with computation
Math ITBS grade equivalent with computation
Mean total A-level problems by May 1, 2013
Mean Reasoning Mind objectives studied by May 1, 2013
22
“SUPPORTED” VS. “UNSUPPORTED”
STUDENTS
During 2012–2013, the majority of second-grade Reasoning Mind teachers in Dallas ISD
were in their second year of Reasoning Mind use and were fully supported during their
first year. A fraction of teachers who taught Reasoning Mind in second grade received
support in 2012–2013, while in some classrooms taught alternatingly by two teachers, one
teacher was supported and the other was not. Students in such classrooms were
considered to have been “half-supported.” The table below presents a summary of
Reasoning Mind metrics as well as ITBS passing rates for the three groups: unsupported,
half-supported, and supported.
As the table on the next page shows,
the supported students had the lowest
passing rates on the pretest, the math
portion of the first-grade ITBS in 2012.
Their passing rate was 50%—5
percentage points—lower than that of
the unsupported students. This
suggests that principals directed
support to the teachers of the least
prepared students. On the posttest,
however, the students of fully
supported teachers saw the greatest
growth in terms of passing rate—8
percentage points, compared to 5
percentage points for the groups with
half-support or no support. They
surpassed the half-support group and
managed to erase some of their gap
with the unsupported students.
23
Table 15. ITBS pre- and posttest results broken down by support level.
No support
(N =9283)
Mean
SD
Half-support
(N =580)
Mean
SD
Full support
(N =994)
Mean
% passing, 2013 math ITBS
(second grade)
60
57
58
% passing, 2012 math ITBS
(first grade)
55
52
50
SD
2013 math score without
computation
168
17
167
18
167
16
2013 math NCE without
computation
49
21
49
22
48
20
2013 math grade equivalent
without computation
2.8
1.0
2.8
1.0
2.7
0.9
2013 math score with computation
169
16
168
16
168
14
2013 math NCE with computation
50
22
49
22
49
20
2013 math grade equivalent
with computation
2.9
0.9
2.8
0.9
2.8
0.8
2012 math score without
computation
149
13
148
14
147
13
2012 math NCE without
computation
48
20
47
21
46
19
2012 math grade equivalent
without computation
1.7
0.7
1.7
0.7
1.7
0.7
2012 math score with computation
150
11
149
12
148
11
2012 math NCE with computation
49
21
49
21
47
20
2012 math grade equivalent with
computation
1.8
0.6
1.8
0.6
1.7
0.6
24
Dallas ISD Reasoning Mind
Results 2012–2013
Time Spent Online Outside
of Regular School Hours
Because Reasoning Mind is an online program, students can use it at any time from any
computer with an Internet connection. This allows some students to study beyond regular
class time. Because students are able to spend time outside of school using Reasoning
mind, a question arises in relation to the district-wide use of Reasoning Mind by second
and third graders in the 2012–2013 school year: how much time did students spend
learning in Reasoning Mind outside of regular school hours?
Overall, students spent 5% of their total time in Reasoning Mind outside of regular school
hours. This resulted in a mean time online for the year of 3.10 for second graders and 2.96
hours for third graders. The amount of time spent online outside of regular school hours
varied quite a bit. Some students logged in for only a few minutes and some worked in
Reasoning Mind for many hours over the course of the year.
Table 16. Descriptive metrics of time spent using Reasoning Mind outside of normal school
hours.
2nd grade
2nd grade
students who
spent at least
an hour online
outside of
school
3rd grade
3rd grade
students who
spent at least
an hour online
outside of
school
7,083
(55%)
3,885
(30%)
7,435
(61%)
4,218
(34%)
Mean time online outside
of school hours
3.10
5.33
2.96
4.90
Time online outside of
school hours minimum
0.01
1.00
0.01
1.00
Time online outside of
school hours maximum
113.27
113.27
169.66
169.66
5.34
6.40
5.61
6.83
Number of Student who
logged in outside of school
hours (and percent of total
population in grade)
Standard deviation for time
online outside of school
hours
25
Dallas ISD Reasoning Mind
Results 2012–2013
Survey Results
A total of 215 principals, assistant principals, and math coaches took a survey regarding
their view of the Reasoning Mind program in Dallas ISD. A further 121 supported teachers
and 424 unsupported teachers took the teacher surveys. Below is a brief summary of their
responses.
VALUE OF REASONING MIND
Table 17. Dallas ISD staff’s desire to continue using Reasoning Mind on their campuses
(administrators) and in their classrooms (teachers).
Would you like to use Reasoning Mind next year?
Yes
Maybe
No
Administrators
64%
24%
12%
Supported teachers
76%
12%
12%
Unsupported teachers
63%
16%
21%
Table 18. Evaluation of change in teacher effectiveness due to Reasoning Mind.
Does Reasoning Mind help teachers be more effective
in the classroom?
Yes
Maybe
No
Administrators
47%
32%
21%
Supported teachers
57%
27%
16%
Unsupported teachers
48%
26%
26%
26
Table 19. Evaluation of benefit of the program to students. Note the difference in the
opinions of teachers with support and those without it.
Do students in my school/classroom benefit from
Reasoning Mind?
Yes
Maybe
No
Administrators
67%
27%
6%
Supported teachers
85%
10%
5%
Unsupported teachers
73%
16%
11%
IMPACT ON STUDENTS
Teachers were asked to evaluate the
impact of the program on their
students. On every question, the
majority of teachers noted
improvement in their students’ skills,
abilities, and mindsets. In general,
supported teachers cited higher
positive impact by approximately 5–10
percentage points on most questions.
The only exception was enjoyment of
mathematics.
27
Table 20. Evaluation of impact on particular student skills, abilities, and mindsets.
What effect does Reasoning Mind
have on students?
Supported
teachers
Unsupported
teachers
Improved
Neutral
Regressed
Reasoning skills
83%
14%
1%
Non-standard thinking skills
76%
23%
1%
Independence in learning
87%
10%
1%
Confidence in mathematical ability
83%
15%
1%
Enjoyment of mathematics
68%
18%
13%
Reading comprehension
62%
30%
1%
Progress in other subjects
61%
39%
0%
Reasoning skills
73%
20%
1%
Non-standard thinking skills
70%
28%
1%
Independence in learning
78%
21%
1%
Confidence in mathematical ability
78%
20%
2%
Enjoyment of mathematics
82%
16%
2%
Reading comprehension
61%
38%
1%
Progress in other subjects
53%
46%
1%
28