DISCLAIMER: This post expresses my personal opinion and experience with ACL Rolling Review (ARR). It does not represent the views of any organisation I am affiliated with.
This post is a critique of the resubmission feature in ACL Rolling Review (ARR) – which must be the core feature of the system. In particular, I will focus on the problem of resubmission when we request the same set of reviewers. I hope that this post will help to raise awareness of the issues with the resubmission process in ARR, and hopefully increase the number of reviewers who follow the guidelines when reviewing resubmissions.
How resubmission should work in ARR
In ARR, when you resubmit a paper, you can request the same set of reviewers as before. This is a feature that is supposed to make the process smoother for authors who have already received feedback from those reviewers and have made changes based on that feedback.
The ARR Reviewer Guidelines state that:
- “You will need to reread the paper and comment on to what extent the authors successfully responded to the previous reviews. Please indicate that you read their revision notes and say whether you feel weaknesses were adequately addressed or not.”
- “In your review, you should refrain from raising new issues that were not discussed in the first round of reviews — otherwise, if resubmissions get a new round of weaknesses identified, then authors will never be able to move forward with commitment.”
Specifically for substitute reviewers, the guidelines state:
- “you should then edit your draft review, where (i) you explain the reasons why the revisions did not adequately address previously identified weaknesses; (ii) you explain the reasons why newly raised issues represent critical issues of soundness; and (iii) you reframe any other newly raised issues as suggestions for improvement.”
Great. So the idea seems to be that reviewers should focus on the changes made by the authors in response to the previous reviews, and not raise new issues that were not discussed in the first round of reviews. Furthermore, another important point is that even substitute reviewers should not raise new issues that were not discussed in the first round of reviews, unless they are critical issues of soundness.
How resubmission actually works in ARR
Unfortunately, the intended process for resubmissions in ARR often diverges from reality. In practice, some (many?) reviewers do not follow these guidelines, raise new issues that were not discussed in the first round of reviews, and assign low scores to the resubmitted paper, even if the authors have made significant changes based on the previous reviews.
Here, ARR offers a good feature to combat this kind of behaviour: Review issue reporting. With this feature, authors can report any issues with the reviews they receive, including issues with resubmissions. A meta-reviewer will then look into the reported issues and need to take them into account when making their recommendation.
However, a significant hurdle remains: the complex nature of conference acceptance. Even if a meta-reviewer acknowledges a reviewer’s deviation from guidelines and provides a constructive evaluation, a paper can still be, or is highly likely to be, rejected because of the initial low scores from reviewers.
One typical example of this is a paper I submitted to ARR, which was resubmitted with the same set of reviewers. All the reviewers raised new issues that were not discussed in the first round of reviews, and one of them even wrote a rewiew that was almost simply a copy of the previous review, without taking into account the changes made. Despite reporting these issues and the meta-reviewer agreeing that the guidelines were not followed (even providing a constructive evaluation with no “Summary of suggested revisions” – implying sufficient revisions), the paper was rejected. This outcome was likely due to the low scores the reviewers assigned.
The absence of a “Summary of suggested revisions” typically signifies that authors have adequately addressed previous concerns. Yet, in this case, the paper was still rejected, leaving us without a clear path forward, effectively leaving the paper “dead in the water” within the ARR system.
Discussion
While I understand that no system is perfect and that ARR keeps improving, I believe that the resubmission process in ARR needs significant improvement. The current system does not effectively address the issues that arise when reviewers do not follow the guidelines, and it can lead to unfair outcomes for authors who have made efforts to improve their work based on previous feedback.
While it might not be a problem with the ARR system per se, I believe that the resubmission process should be more robust and better enforced. If this is not possible, then I would think that the resubmission feature should be removed altogether (i.e. no sticky reviews), as it does not work as intended and can lead to frustration for authors.
In this sense, I also agree that “Experiment with Making Findings Decisions in ARR,” which is proposed in the ACL Peer Review Report, could be another good idea to reduce this kind of problem.