NUANCES OF
CONDUCTING SCIENCE
EDUCATION RESEARCH
Jane Butler Kahle
Condit Endowed Professor, Emerita, Miami University
African Diaspora: Developing Black Scholars in Science
Education for the 21st Century, Part II
June 13-15, 2013 - Atlanta, Georgia
Research in Education
“One striding fact is that the complex world of
education – unlike defense, health care, or industrial
production – does not rest on a strong research base.
In no other field are personal experience and ideology
so frequently relied on to make policy choices, and in no
other field is the research base so inadequate and little
used.”
National Research Council. (1996). Improving student learning: A strategic plan for education research and
its utilization. Washington, DC: National Academy Press.
J.B. Kahle
Most Important Tip…
 Pick a research area and stick with it: EQUITY
 Black/White Achievement Gap: Kahle, Nordland, Charles, & Sims,
1974. An Alternative Education Experience. The Science Teacher, 42 (4),
44-49.
 Girls in Science Education: Kahle & Lakes, 1983. The Myth of
Equality in Science Classrooms, JRST, 20 131-140.
 Equity in System Reform: Kahle, Meece, & Scantlebury, 2000. Urban
African American Middle School Science Students: Does Standards-based
Teaching Make a Difference? JRST, 37 1019-1041.
NUANCE: A SUBTLE DIFFERENCE IN OR
SHADE OF MEANING, EXPRESSION, OR
SOUND.
With thanks and credit to: Patricia B.Campbell, President, CampbellKibler Associates, Inc. & Eric J. Jolly, Director, Science Museum of
Minnesota, PIs, What Works for Whom in What Context: A Practical
Guide to Conducting Evaluations with Diverse Populations, NSF
Grant No. 1146249
More information from: Pat Campbell
<[email protected]>
Beginning Nuances…
Establish TRUST with participants, for example:
Students
Teachers
Principals, Parents, Others
Phrase your research question as a null
hypothesis and see if it still works.
Research Questions
 What factors affect achievement?
 Are there gender differences? If so, what are they?
 How do perceptions of home support, of friends’
interest in science, and of frequency of standardsbased teaching practices differ by gender?
 How are teacher characteristics related to
achievement?
 How do students’ reports of standards-based teaching
positively influence students’ achievement?
Nuances of Data Collection
 Think broadly about what data you may want/need.
 Collect data, including demographic data, widely. You
cannot go back to collect another piece of information
if something of interest pops up during the analysis.
 Disaggregate data by demographic factors that you are
interested in.
 BUT, if you have small numbers, report numerically as
well as percentages.
Nuances of Context
 Context describes the interrelated conditions in which
something (i.e., your research) occurs.
 Context includes the environment, resources,
opportunities, and challenges of your study.
 How participants identify themselves will help the
researcher define the most salient contextual factors.
Demographic Nuances
 Traditional
 New
 Race/Ethnicity
 Sexual Orientation
 Gender
 Culture
 SES (money)
 Citizenship status
 Disability
 Military
Demographics, cont.
 Ask for demographic information ONLY at the end of
an instrument.
 Beginning an instrument with demographic items may
affect participant responses.
 Explain that responses are voluntary and confidential;
provide reasons why you are collecting these data.
 Participants should define their own race/ethnicity and
disability status.
 Indicate that participants may check as many categories
applicable and include ‘other’.
Demographics,cont.
 Review your introductions to data collection to avoid
any ‘triggers’ for stereotyping; for example, don’t
mention if race or gender differences have been found
when an instrument has been used before.
 Determine if the instruments you plan to use have been
validated with a population similar to the one in your
research study.
Demographics,cont.
 For observations and interviews, pre-test the
protocol/rubric with groups similar to those in your
research.
 If feasible, use two observers/interviewers and compare
their findings.
 Watch out for bias! The potential for bias is increased
when the observer/interviewer has different
characteristics that those of the participants.
Nuances of Design
 What intrigues you in the literature, in your work?
 What theoretical construct will you test?
 Do you have access to appropriate subjects?
 Can you establish rapport with the officials involved?
 Can you get IRB (human subjects) clearance?
 Can you find a control or comparison group?
Nuances of Groups
 Be Real! In schools, colleges, or informal education
settings, it is often not feasible to randomly assign
students to experimental and control groups.
 Establish comparison groups using as many
demographic factors as possible; e.g., same school,
grade, class, sex of teacher, type of teacher
certification, length of time teaching, etc.
Nuances of Instruments
 Don’t reinvent the wheel! Use instruments that…
 Have been validated with subjects similar to yours.
 Have been reviewed by members of the target
population.

If not, use more than one instrument to assess a concept and
compare or triangulate the data.
 Check that any instrument used may be easily
responded to by all subjects in your study; e.g,
accessible to subjects with disabilities.
Nuances of Analysis
 Use more than one instrument (e.g., achievement test,
observation protocol, etc.)
 Look for patterns in the data, collected with multiple
instruments. Patterns are especially important when
you have a quasi-experimental, qualitative, or mixed
methods design.
Nuances of Interpretation
 When interpreting demographic differences, consider
possible confounding factors; e.g., SES, family
educational backgrounds, citizenship status, etc.
 If most of your subjects have one demographic
characteristic (e.g., rural), don’t disaggregate on that
characteristic. However, you will not be able to
generalize to other populations.
 Preliminary analyses by subgroups can help determine
disaggregation and aggregation choices.
Interpretations, cont.
 Don’t report findings as ‘approaching statistical
significance’.
 If statistical significance is found, check for effect size.
 Use more than one rater for open ended responses and
establish inter-rater reliability.
 Watch for external (historical) events that may impact
your findings. If they occur,
 Provide alternative explanations for positive or negative
findings.
Nuances of Dissemination
 Prepare a Technical Report – include methods,
subjects, instruments, analyses, and findings;
 Prepare a paper for presentation – add literature
review, discussion (citing supporting and refuting
literature), and summary;
 Present paper at state, regional, or national meeting;
 Polish paper for submission to an appropriate journal –
teaching/research, refereed/non-refereed.
Dissemination, cont.
 Before submitting…
 Prepare Abstract and select Key Words,
 Cross-check Citations with References,
 Ask a Colleague to Review, and
 If needed, get an Editor.
Dissemination, cont.
 Characteristics of Accepted Presentation & Papers
 Clearly stated Research Questions/Hypotheses
 Both Introduction & Discussion grounded in the literature
 Methods appropriate and convincing
 Valid and reliable instruments used
 Analyses appropriate for data and to answer research questions
 Discussion and Summary grounded in the literature
 Clear and concise writing, following APA style
Nuances of Impact
 Professional Impact
 Professional Association Presentations (NSTA, NARST, regional
and state groups)
 Practitioner Journals (Science Teacher, state journals)
 Professional Journals (JRST, Science Education, Science, etc.)
 Books and Book Chapters (Atwater, Russell, & Butler, (in press).
Multicultural Science Education: preparing Teachers for Equity and Social
Justice)
Impact cont.
 Societal
 Newspapers—delay and prepare
 Books and book chapters
 University news releases—same advice
 Legislators, representatives, senators—Pocket
Panoramas
 Superintendents, principals, parents—P2
 Funding agencies—P2, opportunities to present your
research
In closing…
 Many thanks and good luck to all! If I may be of
further help, please contact me at:
 [email protected]