(Last update: 17 January 2023)
I've reviewed for PCSC, NCITE, MJST, PeJARD, and others. Although most of the reviewing are unpaid (aka "service"), it is quite fulfilling to be part of the academic peer review process. Reviewing also lets me know what my peers are currently working on. In this post, I describe how I review a paper.
I don't do any plagiarism check because I assume that the program committee chair/journal editor has performed a "desk check" using current automated tools, like Turnitin, before handing out the paper to the reviewers. I also don't initially read a paper's abstract.
After reading a paper two times, I consider it well-structured and well-written if I can clearly answer the following at a "high level":
- What is the problem/research question being addressed?
- Why is the problem/research question significant or interesting?
- What are existing solutions to the problem?
- What is/are the proposed solution(s)?
- What is the work's main (technical) contribution to the "body of knowledge" in the area?
- Is the contribution "new" and "significant"?
I do a more thorough reading on the third and fourth passes where I take note of the important claims and check the methods, as well as the results, for correctness. I dissect the technical details of the work "as presented" in the paper. The fifth pass is for validating and double-checking my understanding of the paper.
Once I'm confident about my understanding of the paper, I write my own abstract and compare it to the paper's abstract.
I accept a paper based on the following:
- The paper is "relevant" to the conference or journal.
- The contribution is "significant".
- The work is not just a "repeat-experiment" or an "application of existing techniques" on a different dataset or domain. If it is a "repeat-experiment", the "insights" gained from the result are "substantial" and are "extensively" supported by proofs.
- Important claims are supported by citations, proofs, or data analysis and interpretation.
- The "context of the work" is properly "placed" in relation to other work. Relevant papers are properly cited. I should be able to identify the one (or two) reference(s) which motivated the work.
- The paper uses simple words and sentences. Too much use of jargon makes a paper difficult to read and thus defeats the purpose of publication: effective and efficient communication of research results.
- The paper cites scholarly articles (journal articles, conference proceedings, technical reports), not just web sites.
- I believe I can duplicate the experiments conducted and get the same results using the same dataset, environment, and parameters as the authors' (if I have the resources and given enough time). Plus points if the source code and dataset are made public.
- The paper is in the proper format for the conference or journal.
- Tables and figures are readable, can stand on their own, and have useful labels and captions.
The above criteria may seem too strict and rigid. However, I believe peers who review my work have even stricter criteria.
As a final note, I always review with the mindset of accepting a submission. In case I reject a paper, I see to it that my written review describes the things that will make me accept the paper if resubmitted.
