Referee reports
A typical report contains two parts:
First a brief letter to the editor (the cover letter): accept/revise/major revise/reject. (DO NOT
WRITE THIS PART FOR YOUR HOMEWORKS)
Second the report to the editor and to the author.
The structure of a typical report is:
1. Summary of the main propositions:
-question asked
-structure of the model
-main assumptions
-main results
-place in the literature
2. Brief evaluation: is it worth reading? Can it be improved? Is it the right journal?
3. Major comments: what are absolute requirements for revision? The comments must be
detailed enough so that the author knows precisely what he should do.
4. Minor comments: what are the small details that can easily be improved upon? These
points are not related to the scientific quality/acceptability of the paper. Points about style,
grammar, presentation, missing references,… come here.
Below are some useful advices to referees provided by the Canadian Journal of
Economics
(other comments on http://economics.ca/cje/en/referees.php; they add that a typical report
should probably take 3 or 4 hours, but to my experience, this is overwhelmingly
underestimated).
You can also have a look at http://www.sfedit.net/newsletters.htm
1. What is the CJE looking for in a paper?
A potentially publishable paper should meet the following criteria. Failure to meet any one of
these criteria might be sufficient to recommend rejection.
i. The motivation for the paper should be clear and compelling. Typically the motivation
will include a clearly specified research question and a statement as to why this
question is interesting. However, a paper whose primary motivation is to synthesize or
even survey earlier work might also be publishable.
ii. The analysis in the paper should be correct and should be appropriately rigorous given
the research question.
iii. The paper must be sufficiently original to warrant publication. Typically, this
originality arises from new theoretical results or new empirical findings, but it may
arise from new interpretation or synthesis of known material.
iv.
The paper should be well written. In particular, the logical structure of the paper
should be clear, and the paper should be relatively free from errors of grammar and
usage. A skillful author can usually make an intrinsically difficult argument
v.
reasonably easy to follow, while poor writing can make even minor or trivial points
hard to understand.
The paper should be potentially interesting to a reasonably broad group of potential
readers. These readers might be confined to a particular field within economics, but
the ideal paper is one that would capture the interest of other readers as well.
2. Do I need to check the algebra or the empirical analysis?
Only rarely would a referee be in a position to check empirical results. Often the referee has
neither sufficient time nor access to the necessary data to replicate the empirical analysis in a
paper. Usually the best you can do is to convince yourself that the results seem plausible,
given your knowledge of the area, and are internally consistent. The editors are, however,
encouraging authors to make their data and (where relevant) estimation algorithms available
to referees and potentially to other interested parties. Checking algebra is much more feasible
than checking empirical analysis, but even so it may often not be necessary. In particular, if
you are recommending rejection for some reason other than concerns about the correctness of
the analysis, there is no need to check the algebraic details of the paper. Even if you are
favourably disposed toward a paper, it will often be impossible to check everything. One
useful approach is to check some items, particularly the initial model development, and then
to make sure that you can at least understand why the other results make sense. If it is very
hard to follow the algebraic development in a paper, that is in itself a serious flaw in the
paper.
3. What kind of comments are most helpful?
If you are recommending rejection, you should specify the primary reason for the negative
recommendation. If you think the paper is competent but just not interesting enough or not
important enough for the CJE, you should say that. You do not need to find "errors" in the
analysis or "flaws" in the model structure to justify a rejection. For papers where you
recommend acceptance or a revision it is helpful to say what you think the key contribution of
the paper is. It is also very helpful to consider what kinds of condensation might be desirable.
Often a good paper contains some sections and or some pieces of analysis that are less
valuable than others. Ask yourself what parts of the paper are worth publishing. Basically, the
Journal is seeking high value-added per page and this can sometimes be pursued by selecting
only some parts of papers for publication. If a 30 page paper contains a "nugget" of high value
material that can be converted to a 10 page note, it is useful to pass that information along to
the editor. Write a report that will be forwarded to the author and a separate letter that is just
for the editor. You may or may not wish to make your recommendation clear in the report.
However, you should be clear in your letter to the editor as to the nature of your
recommendation (reject, revise, accept). If you are neutral or "close to the margin" on a paper,
that is fine. Just let the editor know where you stand, even if you are standing on the fence.