JASON_EIER_ET_AL_How to Read a Technical Paper
How to Read a Technical Paper
by Jason Eisner (2009)
Multi-pass reading
Skim the paper first, skipping over anything that would take much mental effort. Just get an idea
of where the paper is going, why it was written, what's old hat and what's new to you. To force
yourself to keep moving, give yourself a limited time budget per page or use the autoscroll
feature of your PDF reader.
Now, assuming the paper still seems worthwhile, go back and read the whole thing more
carefully.
Why not practice on this webpage? Go ahead, skim it first.
S. Keshav describes three-pass reading in detail: What are you trying to do on each pass?
Write as you read
Write as you read. This keeps your attention focused and makes you engage with the paper.
Low-level notes
Often it is easiest to scribble notes on the printed-out paper itself, responding in context to the
formulas, figures, and text. In that case, file or scan your annotated copy for future reference.
(Or perhaps annotate the PDF file directly, without printing or scanning. A free alternative to
Acrobat is PDF-XChange Viewer, a Windows program that can also be run on Linux via wine.)
You can use notes on the paper to
restate unclear points in your own words
fill in missing details (assumptions, algebraic steps, proofs, pseudocode)
annotate mathematical objects with their types
come up with examples that illustrate the author's ideas, and examples that would be
problematic for the author
draw connections to other methods and problems you know about
ask questions about things that aren't stated or that don't make sense
challenge the paper's claims or methods
dream up followup work that you (or someone) should do
High-level notes
Low-level notes aren't enough. Also keep high-level notes about papers. It's quite useful to distill
the paper down: summarize the things that interested you, contrast with other papers, and record
your own questions and ideas for future work. Writing this distillation gives you a goal while
reading the paper, and the notes will be useful to you later.
Michael Mitzenmacher writes: "Read creatively. Reading a paper critically is easy, in that it is always easier to tear
something down than to build it up. Reading creatively involves harder, more positive thinking. What are the good
ideas in this paper? Do these ideas have other applications or extensions that the authors might not have thought of?
Can they be generalized further? Are there possible improvements that might make important practical differences?
If you were going to start doing research from this paper, what would be the next thing you would do?"
I suggest sorting your file of notes chronologically, by when you read the paper, since that may
help you find vaguely remembered papers or remember what else you were reading at the time.
Sometimes you'll want to search by author/title/etc., so start the notes for each paper with a
rough citation. (See also How to Organize Your Files.)
If you had to put a lot of effort into really understanding some point, you can share that effort
with others (and record it for your own future reference) by improving the discussion of that
point on the relevant Wikipedia page.
When and where to read
Start early. Leave enough time that if your attention wanders, you can put the paper down and
pick it up again when you're in a better reading mood. This is better than trying to force yourself
through it on a deadline.
Some people find it easier to read at particular times of day, or while eating or walking or riding
an exercise bike. Do you habitually pick up the closest thing to read when you're at the breakfast
table or in the bathroom? Then leave papers there for yourself.
Try reading with a friend! Sit next to each other, looking at the same copy of the paper, and
stay synchronized at the paragraph or sentence level. Read aloud at times. You'll keep each other
moving and help each other through the hard parts. Discuss as you go along.
Set aside time
When you are starting out in a new area, it may take you hours to read a conference paper
thoroughly. That's okay. It's worth spending that much time to really understand a good or
foundational paper. It will pay off in your future reading and research.
I'll never find the time! Don't worry. Not all papers take that long. Many ideas are reused across
papers, so you will get faster at reading. By now, in an area I know well, I can often read a paper
in 30 minutes or less, because the motivation is familiar and I can recognize much of the setup as
standard practice. (After all, most papers fall into an existing tradition. They extend existing
work with one or two genuine new ideas, and some supporting details that may or may not be
significant.)
But I'm already a third-year student. Why is this paper taking me so long? There is no
shame in reading slowly. It still takes me several hours to absorb a paper on something that I
genuinely don't know well. (Also, it takes me hours to review a paper even in my own area,
because the burden is on me to spot all the problems or opportunities for improvement. 75% of
submitted conference papers are rejected, and most of the remaining 25% also need improvement
before publication.)
Which parts to focus on
So do you really have to read the whole paper carefully on your second pass? Sometimes, but not
always. It depends on why you're reading the paper.
I do think that when you are learning a new area, you should read at least some papers extremely
thoroughly. That means knowing what every sentence and every superscript is doing, so that you
really learn all of the techniques used in the paper. And understanding why things were done as
they were: ask yourself dumb questions and answer them. Practice the ability to decode the
entire paper—as if you were reviewing it critically and trying to catch any errors, sloppy
thinking, or incompleteness. This will sharpen your critical thinking. You will want to turn this
practiced critical eye on yourself as you plan, execute, and write up your own research.
However, there will also be occasional papers where it is not worth reading all the details right
now. Maybe the details are of limited interest, or you simply don't feel equipped to understand
them yet. Consider the parts of a typical paper:
Motivation. You'll want to understand this fairly well, or there's no point in reading the
paper at all. But part of the motivation may depend on things you don't know
(mathematical background or past work). If you don't want to chase those references
down now, you could just raise their priority on your reading list.
Mathematics and algorithms. These parts are the technical heart of the paper. So don't
make a habit of skimming them. (You can learn a lot from how the authors solved their
problems.) Nonetheless, you might skim a technical section if
o It seems like an explanation of something you already know. In that case, just
check that it really says what you think.
o While you probably would benefit right away from knowing the method in detail,
this paper is just not a good place to learn it, or it is too advanced for you right
now. Understand what you reasonably can, and then put it on your list of things to
learn for real. Perhaps ask someone else to explain it to you or to recommend a
reading.
o It seems like an ugly ad hoc solution that no one would ever want to use anyway.
The only reason to understand it fully would be if you wanted to criticize it or
improve upon it. (Still, even if you skip the ugly details, understand what the
authors' intuitions were. Think about how to capture those intuitions more
elegantly.)
o It's enough to know for now that the method exists. It seems specialized, so you
might never need it. You'll come back to the paper if you do.
But you should still achieve clarity now about what the method accomplishes (its
interface). Also try to glean when it is applicable, how hard it would be to use,
and what determines its runtime and accuracy. Then you'll remember the method
when you need it.
What you might skip for now are the hard parts: the internal workings of the
method (its implementation) and any proofs of correctness or efficiency.
Experiments. Many papers test their methods empirically. When you're new to a field,
you should examine carefully how this is typically done (and whether you approve!). It
can also be helpful to notice what datasets and code were used—as you may want to use
them yourself in future.
But once you've learned the ropes, you may not always care so much about a paper's
experiments. After all, sometimes you're only reading the paper to stoke your creativity
with some new problems or techniques. I confess that I often pay less attention to the
experimental details—though examples or error analysis do catch my attention because
they often shed light.
If you do care about the conclusions of the paper ("did the method work?" "should I use
it?"), then you should go back and carefully examine the experimental design, including
the choice of data. Were the experiments fair? Do they support the claims? What's really
going on? Are the conclusions likely to generalize beyond this experimental scenario?
In short, invest your time wisely. Focus on what is valuable to take away. If you can't figure out
which parts of the paper are most "interesting"or "important,"do ask someone who should know! If
you don't know who to ask, find other papers that cite this one (via Google Scholar) and see what
they say about this paper.
Delip Rao suggests: "Never read the original paper on X first. Instead read several later papers on what they say
about X, get an idea of X and then read the original paper. Somehow the research community is much better in
explaining ideas clearly than the original authors themselves."
What to read
do creative web search
o experiment with several searches
o put yourself in an author's shoes; what phrases might they have used?
o become a power searcher! (read the help pages for your search engine)
o specifically search at the ACL Anthology, Google Scholar, etc.
track down related work (once you've got a relevant paper)
o backward references: follow the bibliography to earlier papers
o forward references: see who else has cited the work (via an interface such as
Google Scholar)
has someone else already listed the right papers for you?
o survey papers in journals (also called "review articles")
o course syllabi
o reading group webpages
o chapters in textbooks
o online tutorials
o literature review chapters from dissertations
o direct recommendations from friends or professors (perhaps at other institutions)
breadth-first exploration
o read a lot of abstracts (and skim the papers as needed) before deciding which
papers are best to read
o it's okay to read multiple related papers at once, flipping back and forth so that
they clarify one another
o to get a feel for the research landscape in an area, flip through the proceedings of
a relevant recent workshop, conference, or special-theme journal issue
when the going gets tough, switch to background reading
o textbooks or tutorials
o review articles
o introductions and lit review chapters from dissertations
o early papers that are heavily cited
o sometimes Wikipedia
This page online: http://cs.jhu.edu/~jason/advice/how-to-read-a-paper.html
Jason Eisner - jason@cs.jhu.edu (suggestions welcome)
How to Read and Evaluate Technical Papers
Original Text by Bill Griswold
Modified by Gail Murphy (November 18, 1998 and January 6, 2000)
Taken from Rob Walker's web page at University Calgary (January 2004)
Modified slightly by Jonathan Maletic (January 2004 and Nov. 2013)
Reading research papers effectively is challenging. These papers are often written
in a very condensed style because of page limitations and the intended audience,
which is assumed to already know the area well. Moreover, the reasons for writing
the paper may be different than the reasons the paper has been assigned, meaning
you have to work harder to find the content that you are interested. Finally, your
time is very limited, so you may not have time to read every word of the paper or
read it several times to extract all the nuances. For all these reasons, reading a
research paper often requires a special approach.
The first thing to understand is that the research papers you will read have been
published in different kinds of places. Some papers have been published in the
proceedings of a conference. These papers have been peer-reviewed but are
typically strictly limited in length to about 10-12 pages. Other papers have been
published in archival journals. These papers have also been peer-reviewed but
there are typically not any length restrictions, so they generally include more detail
in every dimension. Some papers are technical reports and these are not peerreviewed. In areas related to Computer Science, you may find that there is first a
technical report version which is then later published in a conference or a journal.
If a paper appears both in conference and journal form, the journal version is
typically a later, expanded and revised version of the work.
To develop an effective reading style for research papers, it can help to know what
you should get out of the paper, and where that information is located in the paper.
Typically, the introduction will state not only the motivations behind the work, but
also outline the solution. Often this may be all the expert requires from the paper.
The body of the paper states the authors' solution to the problem in detail, and
should also describe a detailed evaluation of the solution in terms of arguments or
an empirical evaluation (case study, experiment, etc.). Finally, the paper will
conclude with a recap, including a discussion of the primary contributions. A paper
will also discuss related work to some degree. Related work sections put the paper
in perspective with other research being conducted in the area. Papers are often
repetitive because they present information at different levels of detail and from
different perspectives. As a result, it may be desirable to read the paper out-oforder or to skip certain sections. More on this below.
The questions you want to have answered by reading a paper are the following:
What are the motivations for this work? For a research paper, there is an
expectation that a problem has been solved that no one else has published in
the literature. This problem intrinsically has two parts. The first is often
unstated. Think of this as the broader impacts. This is the benefits that are
desired in the world at large; for example, some issue of quality of life, such
as saved time or increased safety. The other is the technical problem, which
is why the people problem does not have a trivial solution; that is, why a
new technological or engineering solution may be required. Implicitly there
is implication that previous solutions to the problem are inadequate.
Occasionally an author will fail to state either point, making your job much
more difficult.
What is the proposed solution? This is also called the hypothesis or idea.
There should also be an argument about why the solution solves the problem
better than previous solutions. There should also be a discussion about how
the solution is achieved (designed and implemented) or is at least
achievable.
What is the evaluation of the proposed solution? An idea alone is usually not
adequate for publication of a research paper.
What argument and/or experiment is made to make a case for the value of
the ideas?
What benefits or problems are identified?
What are the contributions? The contributions in a paper may be many and
varied. Ideas, software, experimental techniques, and area survey are a few
key possibilities.
What are future directions for this research? Not only what future directions
do the authors identify, but what ideas did you come up with while reading
the paper?
As you read or skim a paper, you should actively attempt to answer the above
questions. Presumably, the introduction should provide motivation.The
introduction and conclusion may discuss the solutions and evaluation at a high
level. Future work is likely in the concluding part of the paper. The details of the
solution and the evaluation should be in the body of the paper. You may find it
productive to try to answer each question in turn, writing your answer down. In
practice, you are not done reading a paper until you can answer all the questions.
How to Read an Engineering Research Paper
William G. Griswold
CSE, UC San Diego
Reading research papers effectively is challenging. These papers are written in a very condensed
style because of page limitations and the intended audience, which is assumed to already know
the area well. Moreover, the reasons for writing the paper may be different than the reasons the
paper has been assigned, meaning you have to work harder to find the content that you are
interested in. Finally, your time is very limited, so you may not have time to read every word of
the paper or read it several times to extract all the nuances. For all these reasons, reading a
research paper can require a special approach.
To develop an effective reading style for research papers, it can help to know two things: what
you should get out of the paper, and where that information is located in the paper. First, I'll
describe how a typical research paper is put together.
Despite a paper's condensed form, it is likely repetitive. The introduction will state not only the
motivations behind the work, but also outline the solution. Often this may be all the expert
requires from the paper. The body of the paper states the authors' solution to the problem in
detail, and should also describe a detailed evaluation of the solution in terms of arugments or an
experiment. Finally, the paper will conclude with a recap, including a discussion of the primary
contributions. A paper will also discuss related work to some degree. Because of the repetition in
these papers at different levels of detail and from different perspectives, it may be desirable, to
read the paper ``out of order'' or to skip certain sections. More on this below.
The questions you want to have answered by reading a paper are the following:
1. What are motivations for this work? For a research paper, there is an expectation that a
problem has been solved that no one else has published in the literature. This problem
intrinsically has two parts. The first is often unstated, what I call the people problem.
The people problem is the benefits that are desired in the world at large; for example
some issue of quality of life, such as saved time or increased safety. The second part is
the technical problem, which is: why doesn't the people problem have a trivial solution?
There is also an implication that previous solutions to the problem are inadequate. What
are the previous solutions and why are they inadequate? Finally, the motivation and
statement of the problem are distilled into a research question that can be addressed
within the confines of this particular paper. Oftentimes, one or more of these elements are
not explicitly stated, making your job more difficult.
2. What is the proposed solution? This is also called the hypothesis or idea. There should
also be an answer to the question why is it believed that this solution will work, and be
better than previous solutions? There should also be a discussion about how the solution
is achieved (designed and implemented) or is at least achievable.
3. What is the work's evaluation of the proposed solution? An idea alone is usually not
adequate for publication of a research paper. This is the concrete engagement of the
research question. What argument, implementation, and/or experiment makes the case for
the value of the ideas? What benefits or problems are identified?
4. What is your analysis of the identified problem, idea and evaluation? Is this a good
idea? What flaws do you perceive in the work? What are the most interesting points
made? What are the most controversial ideas or points made? For work that has practical
implications, you also want to ask: Is this really going to work, who would want it, what
it will take to give it to them, and when might it become a reality?
5. What are the contributions? The contributions in a paper may be many and varied.
Beyond the insights on the research question, a few additional possibilities include: ideas,
software, experimental techniques, or an area survey.
6. What are future directions for this research? Not only what future directions do the
authors identify, but what ideas did you come up with while reading the paper?
Sometimes these may be identified as shortcomings or other critiques in the current work.
7. What questions are you left with? What questions would you like to raise in an open
discussion of the work? What do you find confusing or difficult to understand? By taking
the time to list several, you will be forced to think more deeply about the work.
8. What is your take-away message from this paper? Sum up the main implication of the
paper from your perspective. This is useful for very quick review and refreshing your
memory. It also forces you to try to identify the essence of the work.
As you read or skim a paper, you should actively attempt to answer the above questions.
Presumably, the introduction should provide motivation. The introduction and conclusion may
discuss the solutions and evaluation at a high level. Future work is likely in the concluding part
of the paper. The details of the solution and the evaluation should be in the body of the paper.
You may find it productive to try to answer each question in turn, writing your answer down. I
recommend that you keep a notebook on all the papers you read, or mark-up the papers
themselves. You could use my standard two-page form that you can fill out for each paper. In
practice, you are not done reading a paper until you can answer all the questions.
Also, you should be aware of the context of the paper in relation to the other papers in the class.
Often a paper will represent a generalization, new direction, or contradiction to earlier papers.
If you find that filling out this form doesn't work for you, you can try writing a 250 word abstract
of the paper--not rewriting the abstract at the front of the paper, but your abstract, capturing the
above five issues from your perspective. I often find it useful to write an abstract because it
develops the logical connections between the above five issues.
Taking time to writing down questions you have about the paper will often surface thoughts that
were not initially articulated. Perhaps the paper was vague on key issues, or ignored issues that
you think are important. If you come to class with such questions, you are prepared to counter or
preempt my own questions.
Reading a book is somewhat different. Although you want to answer the above questions for a
book, it may not do the book justice given the amount of detail in each chapter. You may want to
fill out the above questions on a chapter-by-chapter basis, and then produce a summary form for
the entire book when you have finished reading it. However, each chapter will have a particular
slant that may make certain questions irrelevant. Also, a book is often not oriented towards
explaining the solution to a research problem. However, engineering books are invariably
oriented towards problem solving of one kind or another.
I have a habit of writing on papers directly, less with books simply because they cost so much. A
well-annotated paper is worth its weight in gold, as it not only contains the content of the paper,
but your assessment of its value to you.
Advice on note taking. Although I have provided a form that can be filled out, I actually
advocate annotating the paper directly. The paper is a rich canvas on which to layer your
thoughts. Here is how I suggest approaching the reading and mark-up process:
Highlight important comments as you go. Using a highlighter, as opposed to underlining,
can really help key senteneces "pop out" at you when you return to review the paper later.
Mark the important paragraphs of the paper according to motivation/problem,
idea/solution, their evaluation, and contributions.
On the front of the paper, write down the take-away message.
On the front of the paper, or near the end, write down your key questions. Other
questions may be written in the margins as you read.
Try to answer the questions for yourself, as best you can. Use Google or other sources as
appropriate.
Until you have been able to complete the above process, it is likely that you have not yet thought
critically enough about the paper. A second pass over the paper is sometimes required to have it
all come together for you. To help you further structure your reading and note-taking activities,
you might want to follow this rubric, using it as a kind of check list
by Jason Eisner (2009)
Multi-pass reading
Skim the paper first, skipping over anything that would take much mental effort. Just get an idea
of where the paper is going, why it was written, what's old hat and what's new to you. To force
yourself to keep moving, give yourself a limited time budget per page or use the autoscroll
feature of your PDF reader.
Now, assuming the paper still seems worthwhile, go back and read the whole thing more
carefully.
Why not practice on this webpage? Go ahead, skim it first.
S. Keshav describes three-pass reading in detail: What are you trying to do on each pass?
Write as you read
Write as you read. This keeps your attention focused and makes you engage with the paper.
Low-level notes
Often it is easiest to scribble notes on the printed-out paper itself, responding in context to the
formulas, figures, and text. In that case, file or scan your annotated copy for future reference.
(Or perhaps annotate the PDF file directly, without printing or scanning. A free alternative to
Acrobat is PDF-XChange Viewer, a Windows program that can also be run on Linux via wine.)
You can use notes on the paper to
restate unclear points in your own words
fill in missing details (assumptions, algebraic steps, proofs, pseudocode)
annotate mathematical objects with their types
come up with examples that illustrate the author's ideas, and examples that would be
problematic for the author
draw connections to other methods and problems you know about
ask questions about things that aren't stated or that don't make sense
challenge the paper's claims or methods
dream up followup work that you (or someone) should do
High-level notes
Low-level notes aren't enough. Also keep high-level notes about papers. It's quite useful to distill
the paper down: summarize the things that interested you, contrast with other papers, and record
your own questions and ideas for future work. Writing this distillation gives you a goal while
reading the paper, and the notes will be useful to you later.
Michael Mitzenmacher writes: "Read creatively. Reading a paper critically is easy, in that it is always easier to tear
something down than to build it up. Reading creatively involves harder, more positive thinking. What are the good
ideas in this paper? Do these ideas have other applications or extensions that the authors might not have thought of?
Can they be generalized further? Are there possible improvements that might make important practical differences?
If you were going to start doing research from this paper, what would be the next thing you would do?"
I suggest sorting your file of notes chronologically, by when you read the paper, since that may
help you find vaguely remembered papers or remember what else you were reading at the time.
Sometimes you'll want to search by author/title/etc., so start the notes for each paper with a
rough citation. (See also How to Organize Your Files.)
If you had to put a lot of effort into really understanding some point, you can share that effort
with others (and record it for your own future reference) by improving the discussion of that
point on the relevant Wikipedia page.
When and where to read
Start early. Leave enough time that if your attention wanders, you can put the paper down and
pick it up again when you're in a better reading mood. This is better than trying to force yourself
through it on a deadline.
Some people find it easier to read at particular times of day, or while eating or walking or riding
an exercise bike. Do you habitually pick up the closest thing to read when you're at the breakfast
table or in the bathroom? Then leave papers there for yourself.
Try reading with a friend! Sit next to each other, looking at the same copy of the paper, and
stay synchronized at the paragraph or sentence level. Read aloud at times. You'll keep each other
moving and help each other through the hard parts. Discuss as you go along.
Set aside time
When you are starting out in a new area, it may take you hours to read a conference paper
thoroughly. That's okay. It's worth spending that much time to really understand a good or
foundational paper. It will pay off in your future reading and research.
I'll never find the time! Don't worry. Not all papers take that long. Many ideas are reused across
papers, so you will get faster at reading. By now, in an area I know well, I can often read a paper
in 30 minutes or less, because the motivation is familiar and I can recognize much of the setup as
standard practice. (After all, most papers fall into an existing tradition. They extend existing
work with one or two genuine new ideas, and some supporting details that may or may not be
significant.)
But I'm already a third-year student. Why is this paper taking me so long? There is no
shame in reading slowly. It still takes me several hours to absorb a paper on something that I
genuinely don't know well. (Also, it takes me hours to review a paper even in my own area,
because the burden is on me to spot all the problems or opportunities for improvement. 75% of
submitted conference papers are rejected, and most of the remaining 25% also need improvement
before publication.)
Which parts to focus on
So do you really have to read the whole paper carefully on your second pass? Sometimes, but not
always. It depends on why you're reading the paper.
I do think that when you are learning a new area, you should read at least some papers extremely
thoroughly. That means knowing what every sentence and every superscript is doing, so that you
really learn all of the techniques used in the paper. And understanding why things were done as
they were: ask yourself dumb questions and answer them. Practice the ability to decode the
entire paper—as if you were reviewing it critically and trying to catch any errors, sloppy
thinking, or incompleteness. This will sharpen your critical thinking. You will want to turn this
practiced critical eye on yourself as you plan, execute, and write up your own research.
However, there will also be occasional papers where it is not worth reading all the details right
now. Maybe the details are of limited interest, or you simply don't feel equipped to understand
them yet. Consider the parts of a typical paper:
Motivation. You'll want to understand this fairly well, or there's no point in reading the
paper at all. But part of the motivation may depend on things you don't know
(mathematical background or past work). If you don't want to chase those references
down now, you could just raise their priority on your reading list.
Mathematics and algorithms. These parts are the technical heart of the paper. So don't
make a habit of skimming them. (You can learn a lot from how the authors solved their
problems.) Nonetheless, you might skim a technical section if
o It seems like an explanation of something you already know. In that case, just
check that it really says what you think.
o While you probably would benefit right away from knowing the method in detail,
this paper is just not a good place to learn it, or it is too advanced for you right
now. Understand what you reasonably can, and then put it on your list of things to
learn for real. Perhaps ask someone else to explain it to you or to recommend a
reading.
o It seems like an ugly ad hoc solution that no one would ever want to use anyway.
The only reason to understand it fully would be if you wanted to criticize it or
improve upon it. (Still, even if you skip the ugly details, understand what the
authors' intuitions were. Think about how to capture those intuitions more
elegantly.)
o It's enough to know for now that the method exists. It seems specialized, so you
might never need it. You'll come back to the paper if you do.
But you should still achieve clarity now about what the method accomplishes (its
interface). Also try to glean when it is applicable, how hard it would be to use,
and what determines its runtime and accuracy. Then you'll remember the method
when you need it.
What you might skip for now are the hard parts: the internal workings of the
method (its implementation) and any proofs of correctness or efficiency.
Experiments. Many papers test their methods empirically. When you're new to a field,
you should examine carefully how this is typically done (and whether you approve!). It
can also be helpful to notice what datasets and code were used—as you may want to use
them yourself in future.
But once you've learned the ropes, you may not always care so much about a paper's
experiments. After all, sometimes you're only reading the paper to stoke your creativity
with some new problems or techniques. I confess that I often pay less attention to the
experimental details—though examples or error analysis do catch my attention because
they often shed light.
If you do care about the conclusions of the paper ("did the method work?" "should I use
it?"), then you should go back and carefully examine the experimental design, including
the choice of data. Were the experiments fair? Do they support the claims? What's really
going on? Are the conclusions likely to generalize beyond this experimental scenario?
In short, invest your time wisely. Focus on what is valuable to take away. If you can't figure out
which parts of the paper are most "interesting"or "important,"do ask someone who should know! If
you don't know who to ask, find other papers that cite this one (via Google Scholar) and see what
they say about this paper.
Delip Rao suggests: "Never read the original paper on X first. Instead read several later papers on what they say
about X, get an idea of X and then read the original paper. Somehow the research community is much better in
explaining ideas clearly than the original authors themselves."
What to read
do creative web search
o experiment with several searches
o put yourself in an author's shoes; what phrases might they have used?
o become a power searcher! (read the help pages for your search engine)
o specifically search at the ACL Anthology, Google Scholar, etc.
track down related work (once you've got a relevant paper)
o backward references: follow the bibliography to earlier papers
o forward references: see who else has cited the work (via an interface such as
Google Scholar)
has someone else already listed the right papers for you?
o survey papers in journals (also called "review articles")
o course syllabi
o reading group webpages
o chapters in textbooks
o online tutorials
o literature review chapters from dissertations
o direct recommendations from friends or professors (perhaps at other institutions)
breadth-first exploration
o read a lot of abstracts (and skim the papers as needed) before deciding which
papers are best to read
o it's okay to read multiple related papers at once, flipping back and forth so that
they clarify one another
o to get a feel for the research landscape in an area, flip through the proceedings of
a relevant recent workshop, conference, or special-theme journal issue
when the going gets tough, switch to background reading
o textbooks or tutorials
o review articles
o introductions and lit review chapters from dissertations
o early papers that are heavily cited
o sometimes Wikipedia
This page online: http://cs.jhu.edu/~jason/advice/how-to-read-a-paper.html
Jason Eisner - jason@cs.jhu.edu (suggestions welcome)
How to Read and Evaluate Technical Papers
Original Text by Bill Griswold
Modified by Gail Murphy (November 18, 1998 and January 6, 2000)
Taken from Rob Walker's web page at University Calgary (January 2004)
Modified slightly by Jonathan Maletic (January 2004 and Nov. 2013)
Reading research papers effectively is challenging. These papers are often written
in a very condensed style because of page limitations and the intended audience,
which is assumed to already know the area well. Moreover, the reasons for writing
the paper may be different than the reasons the paper has been assigned, meaning
you have to work harder to find the content that you are interested. Finally, your
time is very limited, so you may not have time to read every word of the paper or
read it several times to extract all the nuances. For all these reasons, reading a
research paper often requires a special approach.
The first thing to understand is that the research papers you will read have been
published in different kinds of places. Some papers have been published in the
proceedings of a conference. These papers have been peer-reviewed but are
typically strictly limited in length to about 10-12 pages. Other papers have been
published in archival journals. These papers have also been peer-reviewed but
there are typically not any length restrictions, so they generally include more detail
in every dimension. Some papers are technical reports and these are not peerreviewed. In areas related to Computer Science, you may find that there is first a
technical report version which is then later published in a conference or a journal.
If a paper appears both in conference and journal form, the journal version is
typically a later, expanded and revised version of the work.
To develop an effective reading style for research papers, it can help to know what
you should get out of the paper, and where that information is located in the paper.
Typically, the introduction will state not only the motivations behind the work, but
also outline the solution. Often this may be all the expert requires from the paper.
The body of the paper states the authors' solution to the problem in detail, and
should also describe a detailed evaluation of the solution in terms of arguments or
an empirical evaluation (case study, experiment, etc.). Finally, the paper will
conclude with a recap, including a discussion of the primary contributions. A paper
will also discuss related work to some degree. Related work sections put the paper
in perspective with other research being conducted in the area. Papers are often
repetitive because they present information at different levels of detail and from
different perspectives. As a result, it may be desirable to read the paper out-oforder or to skip certain sections. More on this below.
The questions you want to have answered by reading a paper are the following:
What are the motivations for this work? For a research paper, there is an
expectation that a problem has been solved that no one else has published in
the literature. This problem intrinsically has two parts. The first is often
unstated. Think of this as the broader impacts. This is the benefits that are
desired in the world at large; for example, some issue of quality of life, such
as saved time or increased safety. The other is the technical problem, which
is why the people problem does not have a trivial solution; that is, why a
new technological or engineering solution may be required. Implicitly there
is implication that previous solutions to the problem are inadequate.
Occasionally an author will fail to state either point, making your job much
more difficult.
What is the proposed solution? This is also called the hypothesis or idea.
There should also be an argument about why the solution solves the problem
better than previous solutions. There should also be a discussion about how
the solution is achieved (designed and implemented) or is at least
achievable.
What is the evaluation of the proposed solution? An idea alone is usually not
adequate for publication of a research paper.
What argument and/or experiment is made to make a case for the value of
the ideas?
What benefits or problems are identified?
What are the contributions? The contributions in a paper may be many and
varied. Ideas, software, experimental techniques, and area survey are a few
key possibilities.
What are future directions for this research? Not only what future directions
do the authors identify, but what ideas did you come up with while reading
the paper?
As you read or skim a paper, you should actively attempt to answer the above
questions. Presumably, the introduction should provide motivation.The
introduction and conclusion may discuss the solutions and evaluation at a high
level. Future work is likely in the concluding part of the paper. The details of the
solution and the evaluation should be in the body of the paper. You may find it
productive to try to answer each question in turn, writing your answer down. In
practice, you are not done reading a paper until you can answer all the questions.
How to Read an Engineering Research Paper
William G. Griswold
CSE, UC San Diego
Reading research papers effectively is challenging. These papers are written in a very condensed
style because of page limitations and the intended audience, which is assumed to already know
the area well. Moreover, the reasons for writing the paper may be different than the reasons the
paper has been assigned, meaning you have to work harder to find the content that you are
interested in. Finally, your time is very limited, so you may not have time to read every word of
the paper or read it several times to extract all the nuances. For all these reasons, reading a
research paper can require a special approach.
To develop an effective reading style for research papers, it can help to know two things: what
you should get out of the paper, and where that information is located in the paper. First, I'll
describe how a typical research paper is put together.
Despite a paper's condensed form, it is likely repetitive. The introduction will state not only the
motivations behind the work, but also outline the solution. Often this may be all the expert
requires from the paper. The body of the paper states the authors' solution to the problem in
detail, and should also describe a detailed evaluation of the solution in terms of arugments or an
experiment. Finally, the paper will conclude with a recap, including a discussion of the primary
contributions. A paper will also discuss related work to some degree. Because of the repetition in
these papers at different levels of detail and from different perspectives, it may be desirable, to
read the paper ``out of order'' or to skip certain sections. More on this below.
The questions you want to have answered by reading a paper are the following:
1. What are motivations for this work? For a research paper, there is an expectation that a
problem has been solved that no one else has published in the literature. This problem
intrinsically has two parts. The first is often unstated, what I call the people problem.
The people problem is the benefits that are desired in the world at large; for example
some issue of quality of life, such as saved time or increased safety. The second part is
the technical problem, which is: why doesn't the people problem have a trivial solution?
There is also an implication that previous solutions to the problem are inadequate. What
are the previous solutions and why are they inadequate? Finally, the motivation and
statement of the problem are distilled into a research question that can be addressed
within the confines of this particular paper. Oftentimes, one or more of these elements are
not explicitly stated, making your job more difficult.
2. What is the proposed solution? This is also called the hypothesis or idea. There should
also be an answer to the question why is it believed that this solution will work, and be
better than previous solutions? There should also be a discussion about how the solution
is achieved (designed and implemented) or is at least achievable.
3. What is the work's evaluation of the proposed solution? An idea alone is usually not
adequate for publication of a research paper. This is the concrete engagement of the
research question. What argument, implementation, and/or experiment makes the case for
the value of the ideas? What benefits or problems are identified?
4. What is your analysis of the identified problem, idea and evaluation? Is this a good
idea? What flaws do you perceive in the work? What are the most interesting points
made? What are the most controversial ideas or points made? For work that has practical
implications, you also want to ask: Is this really going to work, who would want it, what
it will take to give it to them, and when might it become a reality?
5. What are the contributions? The contributions in a paper may be many and varied.
Beyond the insights on the research question, a few additional possibilities include: ideas,
software, experimental techniques, or an area survey.
6. What are future directions for this research? Not only what future directions do the
authors identify, but what ideas did you come up with while reading the paper?
Sometimes these may be identified as shortcomings or other critiques in the current work.
7. What questions are you left with? What questions would you like to raise in an open
discussion of the work? What do you find confusing or difficult to understand? By taking
the time to list several, you will be forced to think more deeply about the work.
8. What is your take-away message from this paper? Sum up the main implication of the
paper from your perspective. This is useful for very quick review and refreshing your
memory. It also forces you to try to identify the essence of the work.
As you read or skim a paper, you should actively attempt to answer the above questions.
Presumably, the introduction should provide motivation. The introduction and conclusion may
discuss the solutions and evaluation at a high level. Future work is likely in the concluding part
of the paper. The details of the solution and the evaluation should be in the body of the paper.
You may find it productive to try to answer each question in turn, writing your answer down. I
recommend that you keep a notebook on all the papers you read, or mark-up the papers
themselves. You could use my standard two-page form that you can fill out for each paper. In
practice, you are not done reading a paper until you can answer all the questions.
Also, you should be aware of the context of the paper in relation to the other papers in the class.
Often a paper will represent a generalization, new direction, or contradiction to earlier papers.
If you find that filling out this form doesn't work for you, you can try writing a 250 word abstract
of the paper--not rewriting the abstract at the front of the paper, but your abstract, capturing the
above five issues from your perspective. I often find it useful to write an abstract because it
develops the logical connections between the above five issues.
Taking time to writing down questions you have about the paper will often surface thoughts that
were not initially articulated. Perhaps the paper was vague on key issues, or ignored issues that
you think are important. If you come to class with such questions, you are prepared to counter or
preempt my own questions.
Reading a book is somewhat different. Although you want to answer the above questions for a
book, it may not do the book justice given the amount of detail in each chapter. You may want to
fill out the above questions on a chapter-by-chapter basis, and then produce a summary form for
the entire book when you have finished reading it. However, each chapter will have a particular
slant that may make certain questions irrelevant. Also, a book is often not oriented towards
explaining the solution to a research problem. However, engineering books are invariably
oriented towards problem solving of one kind or another.
I have a habit of writing on papers directly, less with books simply because they cost so much. A
well-annotated paper is worth its weight in gold, as it not only contains the content of the paper,
but your assessment of its value to you.
Advice on note taking. Although I have provided a form that can be filled out, I actually
advocate annotating the paper directly. The paper is a rich canvas on which to layer your
thoughts. Here is how I suggest approaching the reading and mark-up process:
Highlight important comments as you go. Using a highlighter, as opposed to underlining,
can really help key senteneces "pop out" at you when you return to review the paper later.
Mark the important paragraphs of the paper according to motivation/problem,
idea/solution, their evaluation, and contributions.
On the front of the paper, write down the take-away message.
On the front of the paper, or near the end, write down your key questions. Other
questions may be written in the margins as you read.
Try to answer the questions for yourself, as best you can. Use Google or other sources as
appropriate.
Until you have been able to complete the above process, it is likely that you have not yet thought
critically enough about the paper. A second pass over the paper is sometimes required to have it
all come together for you. To help you further structure your reading and note-taking activities,
you might want to follow this rubric, using it as a kind of check list