AE Comments for Authors
As you will see from the comments below, you have entirely won over two of the three
reviewers. Reviewers #1 and 2 make some suggestions for revision, mainly stylistic. I
believe that you did a fine job of responding to the comments the reviewers made on the
previous round, and that your paper makes an important contribution to the field.
While I do think that your manuscript could be published as is, I believe that its impact
will be even greater if you choose to make a few relatively minor revisions.
•
•
•
•
•
•
Reviewer #2 believes that you should soften the tone of Section III. I agree
that this is a good idea—one always wants the readers on one’s side. I do not
entirely agree with reviewer #3’s point that you do not need to explain
ethnography, because the IS field understands all about it. It is my opinion
that this description of ethnography is parallel to and balances your preceding
account of statistical generalizability.
I found the openings of the Abstract and the Introduction to be too abrupt and
non-motivational. I’d strongly recommend that you begin both sections with a
sentence (or a few) that properly sets the stage. For example, “Generalizability
is a major concern to those who conduct or consume empirical research
because… . Generalizability is most familiar in the sense of statistical
sampling, but methodologists have long been aware of conceptions of
generalizabiltiy beyond the statistical. The purpose of this paper is to
contribute to… by …” etc.
Somewhere I’d like to see you state that the generalizability claims/arguments
that people make in their papers are in fact hypotheses and hence need to be
tested (you do make this point), not statements of fact. If people simply
accepted a generalizability statement as a hypothesis versus a “proven fact”
some of the worst effects you describe might vanish.
On pp. 13-14, I was bothered by the order of presentation and/or by your
paragraphing. You need to tell us how van Maanen defines first and second
order constructs before you use them in a way that presumes we know what
they are.
Top of page 16, before the new section V. Reading this section, I was again
reminded of an important difference between the two research traditions
which you do not mention and which may contribute to misconceptions or
miscommunications about generalizability. The two traditions use very
different types of concepts/constructs. Neither van Maanen’s level one
constructs (the meanings of subjects) nor his level two constructs (researchers’
interpretations of subjects’ meanings) are the kinds of constructs that most
researchers in the “statistical” tradition use (subjects’ self-report data about
so-called objective observations of reality cast in terms and categories created
by researchers). Consequently, the two groups cannot see generalizability in
the same light. It might be helpful if you made this point.
Page 17 first full paragraph “from theoretical statements to empirical
statements”; isn’t this deduction?
•
One of my comments from the earlier round of reviews you did not take on
board: the comment about typological generalizability. My fault: I didn’t
provide references or explain much. I still think this is an important issue if
you can see fit to address it. Alas, I don’t easily see how it fits your elegant
framework.
One (underappreciated) type of scientific theorizing is the construction of
typologies. When the types are hypothesized as mutually exclusive,
researchers who report their findings are often making generalizability claims
only about a single type (the implication is that findings true of one type
would not be expected to characterize another type). This is different kind of
reasoning from statistical generalizing, because multivariate thinkers assume
that traits are independently distributed in populations; typological thinkers
theorize that traits are not independent and have recurring clusterings of
attributes. A good place to start is McKelvey’s book (although I do not
personally buy his hard-line stance). If you can find a way to deal with this
issue, it would be appreciated (but it is not required).
Author
Title
Publisher
McKelvey, Bill.
Organizational systematics : taxonomy, evolution, classification /
Bill McKelvey.
Berkeley : University of California Press, c1982.
Reviewer # 1
Review of ISR 1999-120-2
The authors have done an excellent job of revising this paper. They have responded to all
the points raised in my previous review.
I am particularly pleased that the authors have re-structured the paper along the lines
suggested by the Associate Editor. It now reads much better. I am also pleased that the
authors have followed my suggestion to discuss generalizability in both interpretivist and
positivistic terms. The imbalance that marked the previous version of this manuscript has
been removed.
Therefore I recommend that this paper should be accepted. I hope the authors agree that
the hard work they put into it was worth the effort. It is now an excellent contribution to
the field.
Minor comments
My only (optional) suggestion is that the authors should consider using simple numbers
(1 to 4) in Figure 3 rather than mnemonic codes. This is a minor stylistic preference on
my part.
Reviewer #2
Review of MS 1999-120 “Generalizing Generalizability in Information Systems
Research”
Strengths
A strength of this paper is that it takes on a controversial topic and provides a masterly
defense of its argument. This is a strength because this paper will educate readers. It
pushes people outside their comfort zone and (to my mind) makes a successful case for
the arguments about broadening the conception of generalizability in IS research. In this
revision the Framework and the example bring home the points being made in the first
part of the paper. I think the ISR readership will benefit from exposure to these ideas,
these arguments, are the way they are framed. I believe the revision successfully
addresses most of the key concerns raised in the previous review. I believe the authors
have faithfully followed the AE’s recommended restructuring of the paper. The focus of
the paper (Section I) is now clear. Section II provides the context that the AE requested.
The case study of Gefen and Straub in Section VI successfully illustrates the framework
established in Section V.
Weaknesses
The one section that I believe needs more work is Section III. In this section the authors
still have a tendency to lapse into lecturing the readers. The goal of educating the readers
should not subsume the research goal of the paper. I found parts of this Section to be
didactic and, at times, bordering on condescending. In the next revision of the paper, the
authors should continue to work hard to remove/recast these parts.
Recommendations
page 6: I think the quote from Rosenberg succinctly makes the point that is essentially
restated in the paragraph above it.
Page 7: I have some issues with the first paragraph in “The Ramifications…” section and,
in particular, with the first section. First, it is overly complex. The clauses ought to be
two separate sentences. Second, I do not understand the point of making particular
reference to “North American research traditions.” If ISR is an international journal why
is this region singled out? I think the authors have not successfully made their point here.
Third, the second clause of the sentence is confusing. To whom are the authors referring?
Themselves? Finally, there is an “axe grinding” tone to the paragraph. I suggest the
authors revisit this paragraph and consider what they are really trying to say in it.
Page 10: The authors should consider moving the final paragraph of section III to the
beginning. It sets the stage for what is to follow.
page 14: the authors explain what ethnography is. I think this is an example of the
didactic tendency in this paper. It should be assumed that ISR readers know what
ethnography is (in the same way we should assume they know what statistics are); if they
don’t they can reference a textbook on the topic.
Overall, I’d like to see this section revisited with an eye to tightening it where possible.
The authors should confine themselves to stating what needs to be said in order to move
their argument forward. But they should avoid the tendency to lecture on the philosophy
of science. This section is a conduit to the heart of the paper – which now begins on page
10. Perhaps if the authors add additional references for those unfamiliar with certain
subject matter, they will be able to achieve their goal of educating the readers without
slowing down the flow and the attention from the primary intent of the paper.
Reviewer #3
Third Review By Reviewer 3: Generalizing Generalizability in IS
Research
Once again, I enjoyed reading a further version of this paper and found it stimulating and
informative. This is an altogether more tightly argued paper that adds considerable clarity
to the subject area and has a strong, consistent line of argument, well backed up by
examples. The paper makes a real contribution to our understanding in IS and is helpful
to researchers at every level in this field. The author(s) (?) is to be congratulated for
taking on board the reviewers’ comments, dealing with these in detail, then doing even
more with the paper. The paper aims at less than before but I think achieves considerably
more because its is much more focused in its intent, and much tighter in its structure and
substance.
I have no objections to this paper proceeding to publication.