The Peer Review Process

The Academic Peer Review Process
Peer Review in This Module
Carrying Out a Detailed Peer Review

The Academic Peer Review Process

Paper is submitted
to conference
Paper
Assigned to AC
Sent to
Reviewers
Reviews
Created
Scores GivenNormally 3/4

AC Summary
Created
Paper Sent
back to Author
Rebuttal
Period
Rebuttal
Submitted
2-Weeks
Based on
Reviews

Re-Scoring Accept/Reject
Rebuttal Sent
to Reviewers
Can go up OR
down

Submitted Accepted % Accepted
2014 2043 465 23%
2015 2120 486 23%
2016 2435 565 23%
2017 2400 600 25%

• Very detailed process with lots of people involved at
each stage
• Relies on the expertise within a group in order to judge
whether a contribution is valid
• Lets reviewers see the level that others are performing at
and adjust their own eﬀort levels

• The significance of the paper's contribution to HCI and
the benefit that others can gain from the contribution:
why do the contribution and benefits matter?
Aspects to Cover

• The validity of the work presented: how confidently can
researchers and practitioners use the results?
Aspects to Cover

• Replicability: how easily would a researcher be able to
carry out this work again based on the methodology
discussion
Aspects to Cover

• Presentation clarity;
Aspects to Cover

• Relevant previous work: is prior work adequately
reviewed?
Aspects to Cover

******* This paper is as good as any top paper in a conference/journal
****** This paper is as good as any standard paper in a conference/journal
*****
This paper may appear in a conference/journal but I wouldn't argue
for it to be there
**** I don't like this paper but wouldn't object to others liking it
*** I would rather not see this paper in a conference/journal
** I would argue for this paper to not be in a conference/journal
*
I would strongly argue for this paper to not be in a conference/
journal

******* This paper is as good as any top paper in a conference/journal
****** This paper is as good as any standard paper in a conference/journal
*****
This paper may appear in a conference/journal but I wouldn't argue
for it to be there
**** I don't like this paper but wouldn't object to others liking it
*** I would rather not see this paper in a conference/journal
** I would argue for this paper to not be in a conference/journal
*
I would strongly argue for this paper to not be in a conference/
journal
Average Cut
Off Point

Paper is submitted
to conference
Paper
Assigned to AC
Sent to
Reviewers
Reviews
Created
AC Summary
Created
Paper Sent
back to Author
Rebuttal
Period
Rebuttal
Submitted
Re-Scoring Accept/Reject
Rebuttal Sent
to Reviewers
Paper Given to
Mike
Sent to Peers/
Classmates
Mike Makes
Comments
Grade Awarded

• Gives you an opportunity to critically evaluate your peers
work and to comment on strengths and weaknesses
• Allows you to understand the level of quality that is
present within the class
• Presents you with an opportunity to learn from others
work

Carrying Out a Detailed Peer Review

The First Read Through
The first read-through is a skim-read. It will help you form an initial
impression of the paper and get a sense of whether your eventual
recommendation will be to accept or reject the paper.
1. What is the main question addressed by the research? Is it relevant and interesting?
2. How original is the topic? What does it add to the subject area compared with other
published material?
3. Is the paper well written? Is the text clear and easy to read?
4. Are the conclusions consistent with the evidence and arguments presented? Do they
address the main question posed?
5. If the author is disagreeing significantly with the current academic consensus, do they have
a substantial case? If not, what would be required to make their case credible?
6. If the paper includes tables or figures, what do they add to the paper? Do they aid
understanding or are they superfluous?

Spotting Major Flaws
While you should read the whole paper, making the right choice of
what to read first can save time by flagging major problems early
on.
1. Drawing a conclusion that is contradicted by the author's
own statistical or qualitative evidence
2. The use of a discredited method
3. Ignoring a process that is known to have a strong influence
on the area under study

Spotting Major Flaws
If methodology is less of an issue, it's often a good idea to look at
the data tables, figures or images first. Especially in science
research, it's all about the information gathered.
1. Insuﬀicient data
2. Statistically non-significant variations
3. Unclear data tables
4. Contradictory data that either are not self-consistent or
disagree with the conclusions
5. Confirmatory data that adds little, if anything, to current
understanding - unless strong arguments for such
repetition are made

Concluding the First Reading
After the initial read and using your notes, including those of any
major flaws you found, draft the first two paragraphs of your
review - the first summarising the research question addressed
and the second the contribution of the work.

This should state the main question addressed by the research and
summarise the goals, approaches, and conclusions of the paper. It
should:
First Paragraph
1. Help the editor properly contextualise the research and add
weight to your judgement
2. Show the author what key messages are conveyed to the
reader, so they can be sure they are achieving what they set
out to do
3. Focus on successful aspects of the paper so the author gets
a sense of what they've done well

This should provide a conceptual overview of the contribution of
the research. So consider:
Second Paragraph
1. Is the paper's premise interesting and important?
2. Are the methods used appropriate?
3. Do the data support the conclusions?

Second Read Through
The Introduction and Literature Review
A well-written introduction:
1. Sets out the argument
2. Summarises recent research related to the topic
3. Highlights gaps in current understanding or conflicts in
current knowledge
4. Establishes the originality of the research aims by
demonstrating the need for investigations in the topic area
5. Gives a clear idea of the target readership, why the research
was carried out and the novelty and topicality of the
manuscript

Second Read Through
Methodology
Academic research should be replicable, repeatable and robust -
and follow best practice.
Methodology should give enough detail so that other researchers
are able to carry out the same research. For example, equipment
used or sampling methods should all be described in detail so that
others could follow the same steps. Where methods are not
detailed enough, it's usual to ask for the methods section to be
revised.

Second Read Through
Results and Discussion
This section should tell a coherent story - What happened? What
was discovered or confirmed?
Discussion should always, at some point, gather all the
information together into a single whole. Authors should describe
and discuss the overall story formed. If there are gaps or
inconsistencies in the story, they should address these and suggest
ways future research might confirm the findings or take the
research forward.

Second Read Through
Results and Discussion
Certain patterns of good reporting need to be followed by the
author:
1. They should start by describing in simple terms what the data show
2. Once described, they should evaluate the trends observed and explain
the significance of the results to wider understanding. This can only be
done by referencing published research
3. The outcome should be a critical analysis of the data collected

Second Read Through
Conclusions
This section is usually no more than a few paragraphs and may be
presented as part of the results and discussion, or in a separate
section.
The conclusions should reflect upon the aims - whether they were
achieved or not - and, just like the aims, should not be surprising.
If the conclusions are not evidence-based, it's appropriate to ask
for them to be re-written.

Second Read Through
References
You will need to check referencing for accuracy, adequacy and
balance.
• Accuracy - Where a cited article is central to the author's
argument, you should check the accuracy and format of the
reference - and bear in mind diﬀerent subject areas may use
citations diﬀerently.
• Adequacy - Are important parts of the argument poorly
supported? Are there published studies that show similar or
dissimilar trends that should be discussed?
• Balance - Check for a list that is helpful for the reader,
doesn’t rely overly on self-citations.

Structuring Your Report
Structure your evaluation of the paper into three sections:
• Summary
• Major Issues
• Minor Issues

Summary
• Give positive feedback first. Authors are more likely to read your
review if you do so. But don't overdo it if you will be recommending
rejection
• Briefly summarise what the paper is about and what the findings are
• Try to put the findings of the paper into the context of the existing
literature and current knowledge
• Indicate the significance of the work and if it is novel or mainly
confirmatory
• Indicate the work's strengths, its quality and completeness
• State any major flaws or weaknesses and note any special
considerations. For example, if previously held theories are being
overlooked

Major Flaws
• Are there any major flaws? State what they are and what the severity
of their impact is on the paper
• Has similar work already been published without the authors
acknowledging this?
• Are the authors presenting findings that challenge current thinking?
Is the evidence they present strong enough to prove their case?
Have they cited all the relevant work that would contradict their
thinking and addressed it appropriately?
• Are there any major presentational problems? Are figures & tables,
language and manuscript structure all clear enough for you to
accurately assess the work?
• Are there any ethical issues? If you are unsure it may be better to
disclose these in the confidential comments section

Minor Flaws
• Are there places where meaning is ambiguous? How can this be
corrected?
• Are the correct references cited? If not, which should be cited
instead/also? Are citations excessive, limited, or biased?
• Are there any factual, numerical or unit errors? If so, what are they?
• Are all tables and figures appropriate, suﬀicient, and correctly
labelled? If not, say which are not

The Peer Review Process

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to The Peer Review Process

Similar to The Peer Review Process (20)

More from Mike Crabb

More from Mike Crabb (20)

Recently uploaded

Recently uploaded (20)

The Peer Review Process