David Williamson Shaffer
Wisconsin Center for Education Research
University of Wisconsin–Madison
Dragan Gasevic
University of Edinburgh
Dear Dragan,
Thank you for forwarding comments on our manuscript “Automating the Detection of Good Reflectionon-Action.” We very much appreciated the thoughtful criticisms and suggestions made by the three referees. We were happy to revise the paper based on their feedback, and we believe that the paper is stronger
as a result. As you requested, we are resubmitting the revised version, with the understanding that it will
be reviewed again before you make a decision regarding publication.
In making our revisions, we noted that one referee believed this “was an interesting paper” and that it
“was well structured and written.” Another noted that it “reads very well, it’s well organised and provides
relevant references to key literature,” and also that it “adequately addresses the theme of the special issue.”
Our strategy, therefore, was to use the feedback to clarify, refine, and strengthen the original argument.
That having been said, the revisions were significant—including both new material based on the referees’
suggestions and significant re-editing for both clarity and style. As a result, the current draft addresses the
concerns raised by the referees in the following manner:
Reviewer A notes that the abstract suggested an examination of how real-world professionals solve complex problems, yet the paper focuses on students. We have revised the abstract and the paper to clarify
that we draw on a theoretical understanding of how experts learn to solve complex problems in order to
explore a technique for automatically detecting one aspect of learning to think like an expert, namely reflection-on-action.
Reviewers A and B both noted a number of missed opportunities to connect our work to more general
work in learning analytics and related fields. This was a particularly helpful criticism. We have revised
the theory section and thoroughly rewritten the discussion to draw clearer connections to existing work on
automated detection of reflection and more general approaches to textual analysis. While thorough comparison to other techniques is beyond the scope of this paper, we have situated our research more clearly
in the field. As Reviewer C observed, there is a lot of work on discourse analysis and natural language
processing. We have expanded our discussion of some of this work, but also note that our goal here is to
explore one relatively simple technique for automated detection of reflection-on-action; future work will
benchmark this technique against other approaches.
Reviewer A also points out that the development of expertise is not necessarily linear, and we have added
this to the discussion on p. 22.
Reviewer A then points out correctly that while we repeatedly used the phrase “good reflection-on-action,”
we never justified how our technique distinguishes good reflection-on-action from poor. This critique was
spot on. We have revised the title and changed the language throughout to indicate that we are automating
the detection only of reflection-on-action, without any claim to the quality of that reflection.
Both Reviewers A and B noted that there was not sufficient information provided in the paper about the
experimental design and the participants. We have added several paragraphs of material to the “Data Collection” section to clarify how the experiment was designed and conducted, who the participants were,
how the groups worked, and so on. In the “Limitations” section, we also discuss some limitations associated with our approach.
All three reviewers noted a lack of clarity in how the coding process was developed and implemented. We
have significantly expanded the sections on coding to detail how the codes were developed, validated, and
implemented.
Reviewer A questioned the usefulness of the first research question given that the finding seemed “obvious.” He or she asked: “Isn’t it anyway essential for a concept-to-concept connection that there is a mentioning of at least X and Y?” We have revised this section to make clearer that our claim is that cooccurrence is sufficient to detect connection, not simply necessary. That is, this finding suggests that cooccurrence implies connection, and because the detection of co-occurrence can be easily automated, the
detection of connection can thus be easily automated.
Reviewer A was also concerned that the definition of “relative domain expert” was inappropriate. We
clarified and justified this terminology in the “Data Collection” section.
Lastly, Reviewer A noted that the “Discussion” section did not sufficiently situate our findings in the literature. We thoroughly revised and extended this section to address this concern and to clarify the “key
message” of the paper. We also included some of the suggested references, which were very helpful.
Reviewer B noted an imbalance between the first half of the paper and the second half, which we have
addressed in several ways. First, we have clarified the research questions and results to make each question clearer and to make the coherence of the three questions clearer. Second, we have significantly expanded the “Discussion” section to clarify the significance of our findings relative to the background and
theory that Reviewer B appreciated.
Reviewer B also suggested that the wording of the research questions made them difficult to understand,
so we revised the language to be clearer and to better show how the questions are connected.
Reviewer C suggested that the theory section needed more diverse references and was too dependent on
Schön. We reviewed this section, and while it is true that we use Schön to anchor the discussion—it is,
after all, his definition of reflection-on-action that we attempt to operationalize—the “Theory” section
(and all of its subsections) includes discussion of and citation of a broad range of sources and perspectives.
As Reviewer B said, the theory section was a strength of the paper. As such, we edited the “Theory” section for clarity and in response to specific criticisms, which we believe makes it clearer and more concise,
but we did not see a compelling reason to significantly expand the number of perspectives included.
However, per Reviewer C’s suggestion, we discuss some of the literature on metacognition.
Reviewer C observed, as did Reviewer B, that the results cover only a small dataset and learning context.
We have significantly expanded the “Discussion” and “Limitations” sections to address this issue and to
detail plans for future research.
Of course, we are delighted that the reviewers found our paper to be “very interesting” and that it “reads
very well.” We found their criticisms both helpful and accurate, and we have endeavored to address them
fully, as we have outlined above. We believe that this substantial revision has strengthened the original
manuscript considerably, and we hope that the current form of the paper is strong enough to warrant publication in the Journal of Learning Analytics.
In closing, we would like to thank the three referees for their thoughtful and valuable feedback. Reviewing manuscripts is a laborious process, and we truly appreciate the obvious care and time that the referees
2
put into their reviews. We have considered carefully their suggestions and feel that they made the manuscript stronger and clearer. We look forward to seeing what they think of the results.
Sincerely,
David Williamson Shaffer
Vilas Distinguished Achievement
Professor of Learning Sciences
University of Wisconsin–Madison
3