Research Seminar – CPIT 695
Reading Skills
1
What is required from you?






how research papers are written,
how to read such papers critically and efficiently,
how to summarise and review them.
how to gain an understanding of a new field, in the
absence of a textbook
how to judge the value of different contributions
how to identify promising new directions
What is a Paper?


A paper: the prime medium for reporting scientific results
In computer science: the conjunction of
 something conceptual (and preferably new),
 something sound, and
 something practical
 that is relevant today (read: IT).
Reading Skills



In principle, a paper provides enough information for its reader to
reproduce its contents: proof, experiment.
The three stages of reading (as one grows up):
 1. the books say blah and thus it is true;
 2. this book says blah;
 3. the author wrote blah.
On reading critically
 Don’t swallow the author’s propaganda, accepting the paper
as the author shaped it: disassemble it to identify its real
thrust; appreciate; probe / question / stress; (if needed)
reassemble: minimize / expand.
Reading Skills



What if you don’t understand something?
What if you think you found a bug?
Recommendation:
 1. Consult people locally.
 2. Send a very polite e-mail to the author (keeping in mind
that you may well be wrong yourself).
 Prudence: You don’t just represent yourself.
 You represent your adviser and your institution.
Some elements for a review






1. Convey your understanding of the paper with a summary.
2. Double up with an analysis.
3. Sum up with an assessment and a recommendation.
4. Add a list of remarks, if any.
Context of the review: Be objective. Be fair. Don’t delay. Be
courteous. Remain confidential.
To summarize
 Reviews should be as comprehensive as possible.
 Reviews should be as courteous as possible.
 Reviews should be as selfless as possible.
Reading Skills



Don't let other people's frame of mind limit your creativity.
Proofreading each other
 What: an investment.
 How: with consideration.
 Be prepared to be misunderstood.
Why read papers
 So you know what’s happening
 Avoid reinventing the wheel


does happen commonly, too many wheels already
Find interesting research topics
Why not to read papers




Cannot read everything
Should not read everything
Can suppress innovation: once you see solutions using a particular theme,
often hard to think differently
Pitfall
 It is of no use to spend almost all of your time in literature review and
seminars.
 It is impossible to ``finish a literature review and then start research."
 New literature is always appearing, and as your depth and breadth
increases, you will continually see new connections and related areas that
must be studied.
 Active listening and reading must be viewed as ``continuing education''
that will involve you for the rest of your career.
 Don't fool yourself into thinking it must be finished before you can begin
research.
Read or not to read, that is the question



Read, of course
Know what’s important
Know what can be ignored without significant loss of
information
What’s in a paper









Abstract
Introduction
Motivation
Problem description
Solution
...
Performance Analysis
Conclusions
Future Work
How to read a paper?
Know why you want to read the paper
 To know what’s going on (e.g., scanning
proceedings)


Papers in your broad research area



title, authors, abstract
introduction, motivation, solution description, summary,
conclusions
sometimes reading more details useful, but not always
Papers you may want to improve on

read entire paper carefully
What to note

Authors and research group


Need to know where to look for a paper on particular topic
Theme of the solution

Should be able to go back to the paper if you need more info

Approach to performance evaluation

Note any shortcomings
Reading Skills







To develop an effective reading style for research papers, it can help to know
what you should get out of the paper, and where that information is located in
the paper.
Typically, the introduction will state not only the motivations behind the work,
but also outline the solution. Often this may be all the expert requires from the
paper.
The body of the paper states the authors' solution to the problem in detail, and
should also describe a detailed evaluation of the solution in terms of arguments
or an empirical evaluation (case study, experiment, etc.).
Finally, the paper will conclude with a recap, including a discussion of the
primary contributions.
A paper will also discuss related work to some degree.
Papers are often repetitive because they present information at different levels
of detail and from different perspectives.
As a result, it may be desirable, to read the paper ``out of order'' or to skip
certain sections.
Questions to be answered while reading a
paper


What are motivations for this work? For a research paper,
there is an expectation that a solution to a problem has been
found that no one else has published in the literature. The paper
should describe why the problem is important and why it does not
have a trivial solution; that is, why a new solution may be
required. Implicitly there is implication that previous solutions to
the problem are inadequate. Occasionally an author will fail to
state either point, making your job much more difficult.
What is the proposed solution? This is also called the
hypothesis or idea. There should also be an argument about why
the solution solves the problem better than previous solutions.
There should also be a discussion about how the solution is
achieved (designed and implemented) or is at least achievable.
Are all concepts and notations introduced before their first usage?
Questions to be answered while reading a
paper



What is the evaluation of the proposed solution? An idea alone
is usually not adequate for publication of a research paper. What
argument and/or experiment is made to make a case for the value
of the ideas? What benefits or problems are identified? Are they
convincing?
What alternative solutions exist? Read a paper critically.
What is your analysis of the identified problem, idea and
evaluation? Is this a good idea? What flaws do you perceive in
the work? What are the most interesting points made? What are
the most controversial ideas or points made? For work that has
practical implications, you also want to ask: Is this really going to
work, who would want it, what it will take to give it to them, and
when might it become a reality?
Questions to be answered while reading a
paper





What are the contributions? How does this contribution relate to work
previously encountered? The contributions in a paper may be many and
varied. Ideas, software, experimental techniques, and area survey are a few key
possibilities.
What are future directions for this research? Not only what future
directions do the authors identify, but what ideas did you come up with while
reading the paper? Sometimes these may be identified as shortcomings or other
critiques in the current work.
What questions are you left with? What questions would you like to raise in
an open discussion of the work? What do you find confusing or difficult to
understand? By taking the time to list several, you will be forced to think more
deeply about the work.
What is your take-away message from this paper? Sum up the main
implication of the paper from your perspective. This is useful for very quick
review and refreshing your memory. It also forces you to try to identify the
essence of the work.
What are the important references cited in this paper by the author?
Reading Skills







Presumably, the introduction should provide motivation.
The introduction and conclusion may discuss the solutions and
evaluation at a high level.
Future work is likely in the concluding part of the paper.
The details of the solution and the evaluation should be in the
body of the paper.
You may find it productive to try to answer each question in turn,
writing your answer down.
In practice, you are not done reading a paper until you can answer
all the questions.
 Use the form (PaperReviewForm.pdf) given to fill OR write
an abstract using the answers from the questions.
Getting the bigger picture




What problem is the paper trying to solve?
 Highlight three to four lines
What are the limitations of prior work?
 Highlight maximum two to three lines
How is the problem solved by this paper?
 What is the paper s’ contribution?
 Highlight three to five lines
How is the proposed solution evaluated?
 What kind of data/experiments were conducted?
 No need to highlight anything; you can highlight two to three
words here
Getting the Details






What problem is the paper trying to solve?
Read the System Model carefully
 The whole paper is going to be based on this model
Understand the gist of the contribution/proposal
 Does it make sense?
 Do you think it will work?
 How will the proposal be evaluated?
Don’t try to read all the math in one go
 Read the assumptions and system model
 Try to work out a solution to the problem
Is the evaluation fair and comprehensive?
 Try to find the next paper you want to read from this section
Understand the results
 Do not miss a single figure and table
 Find the corresponding discussions in the paper and read them thoroughly
Reading Skills – What to retain?




The problem
The basic idea of the proposed solution
Your personal notes on the paper’s mathematics
Shortcomings of the proposed approach
Reading Skills – Ask yourself


What is the message you take away from this paper?
Are you convinced that the paper attempted an important
problem?


Are you convinced that the paper proposed a viable solution?
‰


If your answer is NO, Justify it! If your answer is NO, Justify it!
If your answer is NO, Justify it!
What questions still remain unanswered?
‰
If you find a well-written paper

After reading a well written paper, consider the presentation. It
may be nicely organized, or it may underscore concepts with
thoughtful examples.
 What makes this paper easy to read?
 What level of detail is provided?
 What examples are used to demonstrate important concepts?
 What questions are left unanswered?
 Can the results be generalized?
Getting most out of what you read



Be organized
 Keep an electronic bibliography with notes & pointers to the paper files
 Keep and file all the papers you have read or skimmed
Be efficient - only read what you need to
 Start by reading only the conclusion, scanning figures & tables, and
looking at their references
 Read the other sections only if the paper seems relevant or you think it
may help you get a different perspective
 Skip the sections that you already understand (often the background and
motivation sections)
Take notes on every paper you find worth reading
 What problem are they trying to solve?
 What is their approach?
 How is it different from other approaches?
Getting most out of what you read


Summarize what you have read on each topic - after you have read several
papers covering some topic, note the:
 key problems
 various formulations of the problem they are addressing
 relationship among the various approaches
 alternative approaches
Read PhD theses - even though they are long they can be very helpful in
quickly learning about what has been done is some field. Especially focus on:
 Background sections
 Method sections
 Your advisor's thesis - this will give you an idea for what he/she expects
from you.
Summary in matrix or table form
References











CPIT 695 Notes of Dr. Ahmad Barnawi
Materials to refer:

How to read a CS research paper? – Philip

Paper Review Form
http://www.cs.brandeis.edu/~cs227b/papers/introduction/howToRead.txt
http://wisnet.seecs.nust.edu.pk/people/~khayam/pdf/lecture_research_paper_re
ading.pdf
http://cseweb.ucsd.edu/~wgg/CSE210/howtoread.html
http://www2.cs.uregina.ca/~pwlfong/CS499/reading-paper.pdf
http://cs.au.dk/~danvy/tips-and-tricks/index.html
http://faculty.washington.edu/wpratt/survive.htm
http://www.michbar.org/generalinfo/plainenglish/
“How to have Your Abstract rejected” Mary-Claire van Leunen and Richard
Lipton
26
“Maxims for Malfeasant Speakers” Norman Ramsey