Research
Analysis of
Faculty Peer
Review
Comments
QM Conference 2014
Dr. Andria Schwegler
Dr. Barb Altman
Texas A&M University –
Central Texas
Presentation Agenda
1.
Research Overview and Background
2.
TAMUCT Internal Peer Review Process
3.
Research Design
4.
Coding Scheme and Demonstration
5.
Summary Data of Our Reviewer Comments
6.
Thematic Analysis of Our Reviewer Comments
7.
Potential Solutions to Issues Found – External Reviews?
8.
Potential Solutions to Issues Found – What does Theory Tell Us?
9.
Implications – Revisions to Peer Review Process and Training
10.
Future Research
Schwegler & Altman 2014
QM Conference Presentation
2
1.
1. Research Overview & Background
•
Goal: To empirically examine the content of the peer review comments
elicited in our first round of internal Quality Matters peer reviews at Texas
A&M University Central Texas.
•
Purpose: To better understand what it was about the peer review
comments that led to the most effective improvements in reviewed
courses.
•
Comments were evaluated with three frames in mind:
1.
2.
3.
QM standards of peer review comments that are constructive, specific,
measurable, and balanced.
Qualitative thematic analysis of overall comments.
Theoretical framework of literature on effective feedback for learning
from the cognitive psychology literature.
Schwegler & Altman 2014
QM Conference Presentation
3
2.
•
2. TAMUCT Peer Review Process
Details on our process and a related research project on our faculty
perceptions regarding participating in peer review can be found
here:
Schwegler, A. F., Altman, B. W., & Bunkowski, L. M. (2013). Beliefs
regarding faculty participation in peer reviews of online courses.
Internet Learning, 3, 95-116.
http://www.ipsonet.org/images/Westphalia_Press/Internet_Learn
ing_Journal_2-2/3-1/8.%20Schwegler%20ILJ%203-1.pdf
Schwegler & Altman 2014
QM Conference Presentation
4
2.
•
2. TAMUCT Peer Review Process
An internal peer review process based on the Quality Matters (QM)
framework was initiated over a 1.5 year period ending in Fall 2013.
–
–
–
–
We had to incentivize process to overcome faculty resistance.
We had to keep all reviewers internal to Colleges.
Though faculty were trained on QM, we had to reinforce differences
between design and delivery.
Faculty completed a self-review prior to full review commencing; College
Online Coordinator provided input that was voluntary to correct.
Schwegler & Altman 2014
QM Conference Presentation
5
2.
•
2. TAMUCT Peer Review Process
Across 4 semesters:
– 37 courses were peer reviewed
– 19 different faculty course developers
– 26 additional faculty members did not offer a course for peer
review, but did volunteer to serve as peer reviewers
Schwegler & Altman 2014
QM Conference Presentation
6
2.
•
2. TAMUCT Peer Review Process
Impetus for research:
– Distance Learning Advisory Committee meeting/debrief raised
anecdotal concerns about:
Quality of comments and associated course improvements.
• Lack of learning objectives.
• Hostile exchanges between faculty related to peer reviews and faculty roles.
• Collusion between faculty members who were “buddies” to go “easy” on each
other.
•
–
Subcommittee was formed to evaluate process and whether
improvements justified costs associated with program.
Schwegler & Altman 2014
QM Conference Presentation
7
3. Research Design
•
Study followed protocols for “exempt” archival research.
•
QM recommendations for writing effective comments was used as
primary analysis tool.
•
Associated research questions:
1.
2.
3.
4.
Do the peer reviewers’ recommendations meet QM standards of being
constructive, specific, measurable, sensitive and balanced?
Do the recommendations cite content or information from the course reviewed?
Is specific information from the QM standards quoted or referenced?
Are there identifiable trends in comments (i.e., strengths or weaknesses) that
might inform future trainings?
Schwegler & Altman 2014
QM Conference Presentation
8
3. Research Design
•
Data set from Quality Matters of all peer review comments was deidentified
•
Coding scheme devised using QM guidelines for effective Peer Review
comments:
–
–
–
–
–
–
–
Content from the QM Standards or Annotations
Content from the course under review
Recommendation that is constructive
Recommendation that is specific
Recommendation that is measurable
Recommendation that is sensitive
Recommendation that is balanced
Schwegler & Altman 2014
QM Conference Presentation
9
3. Research Design – Coding Scheme
Coding Rules for Evaluating Comments
Standard/ Annotation Evidence
. = missing data, no comment is provided
0 = comment with no mention of Standard/Annotation
1 = comment with example, phrasing or reference to Standard/Annotation
Course Evidence
. = missing data, no comment is provided
0 = comment with no mention of course content
1 = comment with at least one element from course
Constructive Recommendation
. = missing data, no comment is provided
0 = comment provides no recommendation
1 = comment provides at least one potential recommendation/suggestion or
includes a thought provoking question
Schwegler & Altman 2014
QM Conference Presentation
10
3. Research Design – Coding Scheme
Coding Rules for Evaluating Comments (cont.)
Specific Recommendation
. = missing data, no comment is provided
0 = comment provides only vague or general statements
1 = comment provides at least one example of the recommendation
Measurable Recommendation
. = missing data, no comment is provided
0 = comment with no recommendation
1 = comment with recommendation for an observable suggestion to implement
Sensitive Comment***
. = missing data, no comment is provided
1 = comment includes more negatively-valenced than positively-valenced words
2 = comment includes only neutral words, no emotion provoking words
3 = comment includes more positively-valenced than negatively-valenced words
Schwegler & Altman 2014
QM Conference Presentation
11
3. Research Design – Coding Scheme
Coding Rules for Evaluating Comments (cont.)
Balanced Comment***
. = missing data, no comment is provided
0 = comment points out only weakness/revision
1 = comment points out only strengths/assets
2 = comment points out at least one strength/asset and weakness/revision
Grammatically Correct Sentence
. = missing data, no comment is provided
0 = comment is not a complete sentence
1 = comment is grammatically correct (includes you understood; ignore
punctuation)
Schwegler & Altman 2014
QM Conference Presentation
12
4. Coding Scheme and Demonstration
•
Review Comment Coding Rules Handout
•
Connection for posting your Coding decision
Schwegler & Altman 2014
QM Conference Presentation
13
4. Coding Demonstration – Sample 1
•
Course 4, Standard 4.5 “Very few instructor explanations
or secondary readings.” (Not met)
•
2011- 2013 Standard 4.5: The instructional materials
present a variety of perspectives on the course content.
(1 point)
•
Audience Results
Schwegler & Altman 2014
QM Conference Presentation
14
4. Coding Demonstration – Sample 2
•
Course7, Standard 1.3 “A line about email should be included.
There is a nice statement about writing for an academic
audience, but it doesn’t apply to the online discussions or
email.” (Not met)
•
2011 – 2013 Standard 1.3: Etiquette expectations (sometimes
called “netiquette”) for online discussions, email, and other
forms of communication are stated clearly. (2 points)
•
Audience Results
Schwegler & Altman 2014
QM Conference Presentation
15
4. Coding Demonstration – Sample 3
•
Course 11, Standard 3.5 “Might want to consider doing
some review quizzes.” (Met)
•
2011-2013 Standard 3.5 Students have multiple
opportunities to measure their own learning progress.
•
Audience Results
Schwegler & Altman 2014
QM Conference Presentation
16
4. Coding Demonstration – Sample 4
•
Course 23, Standard 2.1 “Suggest the knowledge objective
(improve) be revised and made more measureable.” (Met)
•
2011- 2013 Standard 2.1 The course learning objectives
describe outcomes that are measurable. (3 points)
•
Audience Results
Schwegler & Altman 2014
QM Conference Presentation
17
4. Coding Demonstration – Sample 5
•
Course 29, Standard 8.2 “Not much auditory.” (Not met)
•
2011- 2013 Standard 8.2 The course contains equivalent
alternatives to auditory and visual content. (2 points)
•
Audience Results
Schwegler & Altman 2014
QM Conference Presentation
18
4. Coding Demonstration – Wrap-Up
•
And you think those were challenging, try this one!!
•
Course 26, Standard 3.4 “I’m not sure how the sequencing
work (sic) in this course. But I’m not sure it is needed
either.” (Not met)
•
2011-2013 Standard 3.4 The assessment instruments
selected are sequenced, varied, and appropriate to the
student work being assessed.
Schwegler & Altman 2014
QM Conference Presentation
19
5. Summary Data
•
3034 – Total Number of Reviewer Comments Possible
•
925 – Peer Review Comments Provided (30.49%)
•
2109 – Missed Opportunities to Comment (69.51%)
Schwegler & Altman 2014
QM Conference Presentation
20
5. Summary Data
•
Missing data was distributed throughout courses, except
Course 15 – no comments at all
– Course 35 – one comment total
– Course 11 and Course 30 – two comments total
– Course 34 – three comments total
– Course 24 – five comments total
– Course 14 – six comments total
–
Schwegler & Altman
2014 QM Conference Presentation
21
5. Summary Data
•
Missing data was distributed throughout reviewers, except
Reviewers who provided no comments: 7 courses
– Reviewers who provided one comment: 4 courses
– Reviewers who provided two comments: 6 courses
– Reviewers who provided three comments: 1 course
–
•
Some reviewers provided extensive comments
–
–
Reviewer provided 41/41 comments for Course 20
Reviewer provided 40/41 comments for Courses 2, 8, 31
Schwegler & Altman
2014 QM Conference Presentation
22
5. Summary Data by Code
Code
0
Standard Course
Construc Specific Measure Sensitive
Evidence Evidence tive
able
268
112
615
714
740
---
Balanced Sentence
Form
239
189
1
657
813
310
211
185
6
480
736
2
---
---
---
---
---
800
206
---
3
---
---
---
---
---
119
---
---
Schwegler & Altman
2014 QM Conference Presentation
23
5. Data-Based Answers to Research Questions
•
Do the peer reviewers’ recommendations meet QM standards of being
constructive, specific, measurable, sensitive and balanced?
–
•
Do the recommendations cite content or information from the course
reviewed?
–
•
Generally, yes.
Is specific information from the QM standards quoted or referenced?
–
•
Generally, no.
Generally, yes.
Are there identifiable trends in comments (i.e., strengths or weaknesses)
that might inform future trainings?
–
OH YES!
Schwegler & Altman
2014 QM Conference Presentation
24
6. Thematic Analysis of Comments:
Strengths and Weaknesses
•
Faculty members willing to give each other “benefit of the doubt”
even though does not follow process
– Situations where evidence was not found yet course determined
to meet standard
–
–
Course 28, Standard 1.5, Reviewer 2 “I could not find a place where
prerequisites are stated on the syllabus or in Blackboard. Dr. X does address a
number of kinds of required competencies, so I have no problem saying this
standard is met. It might be good to add course prerequisites, though.”
(Assigned “Met”)
Course 9, Standard 1.3, Reviewer 3 “I think when looking at the course, it
just depends on the perspective of the reviewer. Dr. X doesn’t from what I can
see state “etiquette” rules but within his instructions, he tells the student what
the expects.” (Assigned “Met”)
Schwegler & Altman 2014
QM Conference Presentation
25
6. Thematic Analysis of Comments:
Strengths and Weaknesses
•
Lack of module level objectives
– Either not found or professor deliberately chooses not to include
them
Course 6, Standard 2.2, Reviewer 1 “Dr. X has made a good argument for the lack
of module learning objectives being explicitly stated.” (Assigned “Met”) AND
Reviewer 3 “I mark this met even though technically it’s not accurate. I accept and
believe in Dr. X’s assertions about modular objectives in this level of course. They can
inhibit the imagining of creative alternatives.” (Assigned “Met”)
– Course 27, Standard 2.2, Reviewer 1 “ There are no module/unit learning
objectives. All assignments are geared toward accomplishment of the overall course
objectives. One could argue that there are 38 units (Dr. X calls these “blocks” in the
running notes section), and that perhaps each unit should have specific learning
objectives attached. Some of these blocks do ask the students to do things (e.g.,
reflect, describe, practice), so these could be considered learning objectives that are
not clearly stated as such. Perhaps providing a simple learning objectives statement
for each block would solve this problem? “(Assigned “Met”)
–
Schwegler & Altman 2014
QM Conference Presentation
26
6. Thematic Analysis of Comments:
Strengths and Weaknesses
•
Confusion in Distinguishing between Design and Delivery
– Reviewers frame comments related to the specific professor as
opposed to the course
–
–
Course 6, Standard 4.5, Reviewer 3 “One of Dr. X’s strengths as a teacher is
that he values quality of argument over rigid agreement with his own point of
view – something I consider essential in academia.”
Course 20, Standard 1.1, Reviewer 1 “I am impressed with the clarity of
what Dr. X does to meet this standard.” Standard 2.2 “I am impressed with
the detail Dr. X uses for his discussion items and for his writing assignments.”
Standard 3.1 “I am impressed with Dr. X’s use of multiple choice quizzes as
supplements to his heavy writing requirement.” Standard 4.2 “I am
particularly impressed with Dr. X’s use of oral reinforcement of written
materials.” Standard 5.2 “I particularly like Dr. X’s requirement that students
respond to each other’s discussion posts.”
Schwegler & Altman 2014
QM Conference Presentation
27
6. Thematic Analysis of Comments:
Strengths and Weaknesses
•
Not Taking Student’s Perspective
– Comments related to media access demonstrate where reviewer
“took the blame” rather than advocating for students; final
comment does have a recommendation
Course 19, Standard 6.4, Reviewer 2 “The link to the films would not work for
me when I tried but there may be a problem on my end!” (Assigned “Met”)
– Course 27, Standard 1.1 Reviewer 1 “I like Dr. X’s introduction and course
overview – these help set the tone for the class. Both audio and readable formats
are provided – this is good. I could not pull up the .mp3 file, but I think my
computer is just being slow.” (Assigned “Met”)
– Course 36, Standard 6.4, Reviewer 3 “The mp3 files would not open on my Mac.
It could be that the files are large, and I did not wait long enough. If that is the case
(i.e., large files create longer downloads), then stating as much might reduce
student frustration.” (Assigned “Met”)
–
Schwegler & Altman 2014
QM Conference Presentation
28
6. Thematic Analysis of Comments:
Strengths and Weaknesses - Hints
•
Issues/concerns that arose but not to level of full theme
1.
Confusion on copyright and fair use
2.
Universal design standards (unclear how to evaluate for Standard 8)
3.
Institutional policies on standards for course design (particularly as relates to
the appropriate level objectives and assignments in a undergraduate vs.
graduate classes)
Schwegler & Altman 2014
QM Conference Presentation
29
6. Thematic Analysis of Comments:
Strengths and Weaknesses
•
Strengths – some faculty took the process very seriously and wrote
very effective comments!
•
Course 2, Standard 2.2, Reviewer 3 “As stated in the annotations for module/unit
learning objectives, ‘…these objectives are prominently stated in the corresponding
module or unit so that they are accessible to the student from within the online
classroom.’ The module level objectives are stated on the Course Information page, on
the syllabus, and on the learning module tab, but they are not nested within each
learning module to provide students with focused directions as they work on specific
content. For example, I see for Module 1 that I am to “conduct and apply the statistical
foundations…” but I’m not sure exactly what I should be getting out of the Lecture 5 link
specifically. The addition of objectives for each lesson in the Learning Module would
make the targeted objectives clear for each set of content. I noticed that for objectives
2.151 and on (on the Course Information page; the order is different on the syllabus) you
used action verbs to describe what students would do with the content (e.g., calculate
confidence intervals…, conduct hypothesis testing…). This level of specificity is helpful.
Add such specificity to all objectives. For example, when listing “qualitative” and
“measures of central tendency,” it is not clear what students need to do with the
information (perhaps “define and recognize” and “calculate,” respectively?).”
Schwegler & Altman 2014
QM Conference Presentation
30
6. Thematic Analysis of Comments:
Strengths and Weaknesses
•
Another example of an exemplar comment:
•
Course 31, Standard 2.2, Reviewer 2 “Stating the module level objectives
on the weekly modules is helpful for students. On the surface, the module
level objectives are well-stated. They describe specific outcomes regarding
content and lead with behavioral verbs. However, the module-level objectives
need revision regarding alignment as explained in my comments for Standard
3.1. (Given the discrepancy of wording for module level objectives between the
Course Content links and the syllabus, have the objectives stated on the
syllabus already been revised as I recommend below?) In addition, being naive
to course content, it is not clear how each set of module level objectives maps
onto the broader course level objective. Numbering the course level objectives
and adding the relevant number to the end of each module level objective
would help students understand the connection between the module level and
course level objectives.”
Schwegler & Altman 2014
QM Conference Presentation
31
7. Potential Solutions – External Reviews?
•
Internal vs. External Peer Reviews: Would “Official” QM Reviews
Address Found Issues?
–
–
–
•
In an official review, the Master Reviewer as Chair would provide more
direction and peer coaching for newer peer reviewers not skilled in
writing comments or that do not fully understand design/delivery
distinction.
In an official review, the review would not proceed without module
level objectives.
In an official review, other reviewers would not know and have a
relationship with course developer and perhaps better maintain
objectivity (e.g. no granting "benefit of the doubt," decisions would
strictly be made on course content).
Schwegler & Altman 2014
QM Conference Presentation
32
7. Potential Solutions – External Reviews?
•
But
–
–
–
–
Official reviews would not provide a feedback loop to connect us with
the needs of our faculty.
Internal reviews help us identify the training needs of our faculty.
Internal reviews help our faculty practice distinguishing between
design and delivery.
Internal reviews provide feedback on the peer review process that
comes from performing the task itself instead of coming from other
people (i.e., task-generated feedback). Feedback implicit in the task is
an important source of feedback that has implications for learning –
in addition to feedback provided by others.
Schwegler & Altman 2014
QM Conference Presentation
33
8. Potential Solutions: What does Theory
Tell Us?
•
Acknowledging Our Goals for Peer Review
Improvement of online course offerings
– Faculty buy-in of process
–
•
Distinguishing Our Goals during the Review Process
–
–
•
Attempting to achieve both goals may have hampered our efforts to obtain
them
Uncoupling our goals may help us better achieve them in the future
Integrating Lessons from Feedback Intervention Theory
Schwegler & Altman 2014
QM Conference Presentation
34
8. Potential Solutions: Feedback Intervention
Theory
•
Kluger &DeNisi (1996). The effects of feedback interventions on
performance: A historical review, a meta-analysis, and a
preliminary feedback intervention theory. Psychological Bulletin,
119(2), 254-284.
•
“We argue that a considerable body of evidence suggesting that
feedback intervention (FI) effects on performance are quite variable
has been historically disregarded by most FI researchers. This
disregard has led to a widely shared assumption that FIs consistently
improve performance” (p. 254).
Schwegler & Altman
2014 QM Conference Presentation
35
8. Potential Solutions: Feedback Intervention
Theory
•
Feedback Interventions (FI) are “actions taken by external agents to
provide information regarding some aspect of one’s task
performance” (Kluger & DeNisi, 1996, p. 255).
•
Though the theory is most readily applied to teachers providing
feedback to students on coursework, the definition is consistent
with QM peer reviewer comments to course developers.
Schwegler & Altman
2014 QM Conference Presentation
36
8. Potential Solutions: Feedback Intervention
Theory
Self
Affect
Cognitive Resources
FI Cues
Focal
Task
Motivation
Performance
Non-Focal
Tasks
Task
Details
Task
Details
Learning
Schwegler & Altman
2014 QM Conference Presentation
37
8. Integrating Theory with Data
Is task
easy?
YES
Self Goals
Self
Self-esteem,
Control,
Impression
management
Is focal task
important
for self
goals?
Performance
Enhancement
Performance
Interference
NO
Reduce
Performance
Schwegler & Altman
2014 QM Conference Presentation
38
8. Integrating Theory with Data
•
Comments targeting self goals that may have hampered
performance:
•
Course 5, Standard 2.4, Reviewer 1 “You do a terrific job of connecting
specific assignments to particular learning objectives. Good work!”
•
Course 29, Standard 6.3, Reviewer 3 “One of the better courses I have
seen!”
•
Course 28, Standard 2.5, Reviewer 3 “Dr. X does a terrific job keeping
the objectives in front of her students on every level of the course.”
Schwegler & Altman
2014 QM Conference Presentation
39
8. Integrating Theory with Data
NO
Attain other
self goals?
YES
Increase
Effort
FeedbackStandard None
Discrepancy
Maintain
Effort
Positive
Focal
Task
Reduce
Effort
Negative
Effort reduce YES
discrepancy?
NO
Schwegler & Altman
2014 QM Conference Presentation
Maintain or
Increase
Effort
Learning or
Self
40
8. Integrating Theory with Data
192 “Not Met” Ratings (20.76%) by QM Rubric Standard
35
30
25
20
15
10
5
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
2.1
2.2
2.3
2.4
2.5
3.1
3.2
3.3
3.4
3.5
4.1
4.2
4.3
4.4
4.5
4.6
5.1
5.2
5.3
5.4
6.1
6.2
6.3
6.4
6.5
7.1
7.2
7.3
7.4
8.1
8.2
8.3
8.4
0
Schwegler & Altman
2014 QM Conference Presentation
41
8. Integrating Theory with Data
Temporary
Performance
Interference
Interruption
YES
Task
Details
Hypotheses
correct?
Is the task
well
known?
YES
NO
Generate
and test
hypotheses
Hypotheses
match
reality?
Schwegler & Altman
2014 QM Conference Presentation
Positive
Learning
No or
Negative
Learning
NO
Quit
42
8. Integrating Theory with Data
•
Feedback that is constructive, specific, and measurable should
direct/maintain attention at task detail level.
•
Comments that help course developers test hypotheses:
–
Course 14, Standard 1.8, Reviewer 1 “I do see that Dr. X has students introduce
themselves through course content and I can accept this as a legitimate
introduction... I share Dr. X’s worry about the redundancy of student
introductions in my classes. I reluctantly incorporated introductions in my classes
two or three semesters ago, but did so as a first attempt for students to think
through the over-arching questions of the class. I was pleased with how this
worked. Students began selling the online experience and the benefit of taking
“my” classes... There might be a way to let students “introduce” themselves to
others and you… This is what I have tried to do in a sly way.” (Assigned “Not
Met”
Schwegler & Altman
2014 QM Conference Presentation
43
8. Integrating Theory with Data
•
Comments that refute hypotheses for the course developer are likely to
be ineffective.
•
Comments that do NOT help course developers test hypotheses:
Course 5, Standard 8.2, Reviewer 3 “This is something that, as a university, we
need to address…it is also something that faculty should not be required to be
responsible for. That being said, I did not see any mention of alternative formats for
content being available.” (Assigned “Not Met”)
– Course 9, Standard 1.8, Reviewer 1 “I do not see a place where students are asked
to introduce themselves. I suspect they ‘introduce’ themselves, though, in the ways
that are most meaningful (through discussion). I can’t remember ever introducing
myself in a graduate class.” (Not met
– Course 24, Standard 1.8, Reviewer 1 “Dr. X, I know that we have talked about this
already but you may want to include something to satisfy this requirement for the
sake of QM but this is up to you of course.” (Assigned “Not Met”)
–
Schwegler & Altman
2014 QM Conference Presentation
44
8. Theoretical Implications
•
Comments focused on task details that help course developer test
hypotheses are most likely to promote positive learning.
–
Empirical question: Are these types of comments equally effective for
positive and negative feedback-standard discrepancies?
•
•
QM is moving toward recommending (and requiring) comments for all standards
regardless of “Met” or “Not Met” verdict. How helpful is this?
Positive feedback to course developer for participation should be
separated from constructive comments directed to course
improvement.
–
Empirical question: Are the sensitive and balanced recommendations for
writing effective comments distracting? Do these comments need to be
shifted from asynchronous to synchronous contact only?
Schwegler & Altman 2014
QM Conference Presentation
45
9. Practical Implications - Revisions to Peer
Review Process
•
So, choice between internal vs. external reviews going forward is
not an easy one
•
Internal reviews are still more palatable to the majority of faculty,
although this differs by College
–
•
One College will be “experimenting” with external reviews this coming year
Primary discussion becomes how to proceed with a revised internal
review process that would have better safeguards for improved
comments and, therefore, course improvements
Schwegler & Altman 2014
QM Conference Presentation
46
8. Practical Implications – Revisions to Faculty
Training
•
Training Going Forward
–
The APPQMR these faculty took was prior to changes that “beefed up”
sections on writing effective peer review comments.
•
–
Training needs to provide a much better distinction between design and
delivery.
•
–
Provide a refresher training with writing peer review comments emphasized with lots
of examples
New Course Development Program where faculty work more directly with an
Instructional Designer has been adopted and clearly shows distinction
Training needs to emphasize importance of module level objectives and
taking the student perspective.
•
New Course Development Program requires module level objectives and includes
questions regarding student perspective
Schwegler & Altman 2014
QM Conference Presentation
47
8. Practical Implications – Training
•
Training going forward (cont.)
–
Training needs to be offered on
• Universal design
• Copyright and fair use
Schwegler & Altman 2014
QM Conference Presentation
48
9. Future Research
•
Peer Review Coding
–
Comparative study of other institutions doing Internal Reviews
•
–
Study of QM Official Reviews
•
•
–
•
Would highlight whether issues we found are unique to TAMUCT or more widespread
Would highlight whether these same issues or different ones exist with official reviews
Would help QM further refine process and training
Comparison of training programs to determine which training ameliorates
the issues identified
Tests of Feedback Intervention Theory in Peer Review Context
Schwegler & Altman 2014
QM Conference Presentation
49
Research Analysis of
Faculty Peer Review
Comments
QM Conference 2014
Dr. Andria Schwegler
Dr. Barb Altman
Texas A&M University –
Central Texas
Questions?
Contact information:
[email protected]
Thank you for attending!