It’s Time We Fix the Peer Review System

• peer review
• publishing
• reviewing
• scholarship

Peer review has been described as the “backbone” or “cornerstone” of the scholarship process.^1,2 Feedback provided to authors and editors ensures that conclusions put forward serve to advance knowledge in a discipline. Authors are given the opportunity to incorporate the feedback into their manuscripts with the goal of enhancing quality and ensuring the information is useful to the target audience. Unfortunately, the peer review system as it exists is broken. A recent editorial by Dean and Forray (both editors for a management theory and practice publication) described the peer review system as “unsustainable,” noting that while publications are viewed as academic scholarship, peer review, a fundamental and non-negotiable component to the production of that scholarship, is deemed “invisible” in terms of recognition or reward.³ Grinker and Besnier (also journal editors) argued in a 2016 editorial that the “depersonalization” of the publication process (ie, transitioning away from paper and postal mail communication and toward email and computer-driven communication) and dramatic expansion in the number of journals have led to a decline in development of professional relationships between reviewers and editors that previously sustained the balance between publishing and reviewing.⁴ Authors and faculty members are receiving more invitations to review as more manuscripts are being submitted to a growing number of scholarly publications, with editors now often sending out 10-12 invitations (or more) to capture 1-2 reviewers.⁴ A recent commentary by an editor of the American Journal of Health-System Pharmacy also noted the challenges in recruiting peer reviewers with specific areas of expertise, and in using the advice offered by reviewers whose qualifications or knowledge about a topic may be unknown.⁵

Some in the field have identified a lack of clear incentives to provide peer reviews, calling for increased recognition or reward, or perhaps even modest monetary compensation.^6-8 While the current model relies on “faculty citizenship” for its survival, it is not “transactional” in nature, meaning the peer reviewer is not intended to serve as a stand-in consultant scientific writer or in some other traditionally compensated role. Any model for incentivizing peer reviewers through material means would need broad support among all stakeholders and a clear purpose, goals, and outcomes for its success. Simply paying peer reviewers for their services in a transactional way with no feedback or recognition is not the solution.

One of the major challenges involved in giving credit for peer reviewing activities in terms of promotion and tenure (P&T) is an inability to evaluate quality, whether referring to the quality of the review itself or the contribution of the review to the overall goals of a particular journal. Unlike metrics regarding the quality of scholarship (eg, impact factor, h-index), no such standardized metrics exist for evaluating the quality of peer reviews provided. A new company, Publons (http://www.publons.com/), is trying to change that paradigm.^9,10 Using Publons, journal editors, peer reviewers, and authors can create a profile that displays their contributions to the publication process. Publons allows users to submit information about their publications, journal affiliations, and reviews, which it then verifies. Users can rate the quality of publications, editors can endorse reviews, and “level up” their achievements, creating a type of “gamification” of the scholarship and peer review process. However, the service is not without controversy.¹¹ In general, reviewers are not allowed to share the content of reviews (ie, post the content to Publons) without permission from the editors and authors.¹² The Committee on Publication Ethics (COPE) has published ethical guidelines for peer reviewers stating that the peer review process should remain confidential unless all parties give permission.¹³ It remains to be seen how Publons will adapt its model to already existing structures and rules set by editors and publishers.

There are a number of changes and modifications to the current system that would address common challenges facing scholarly journals, many of which were outlined by Haines and colleagues in a recent commentary.¹⁴ In a companion piece in the same journal, Baker and colleagues identified ways that authors could assist in improving the peer review system.¹⁵ The types of changes required will need to involve all stakeholders, including journal publishers, editors, authors, reviewers, and even readers.

Journal publishers and editors should conduct surveys to determine why reviewers accept (or decline) invitations to review, both at the general level (meaning whether a reviewer chooses to review for a particular journal at all) and at the individual manuscript level. One of the most important factors identified by the research to-date of why reviewers accept an invitation is alignment with the potential reviewer’s area(s) of interest.¹⁶ Because the American Journal of Pharmaceutical Education’s manuscript management system uses “classification matches” (ie, matching a personal classification to the classification of the manuscript) to determine potential interested parties, reviewers should ensure their interests are up-to-date in the system to avoid receiving invitations that are outside of their desired topics or expertise. Reviewers also should ensure that their email addresses are up-to-date in the manuscript management system to avoid missed opportunities to review manuscripts that may be correctly aligned with their interests.

Editors should increase efforts to acknowledge and show appreciation for peer reviewers, including publishing an annual list of all reviewers for each volume, or acknowledging those who provided peer reviews at the individual manuscript or issue level. A 2015 study of 2,982 reviewers sponsored by the journal publisher Wiley found that receiving acknowledgment from a journal was the second most valued type of reward that could be offered for reviewing, ranking just behind feedback on the review quality.¹⁷ The Journal has made strides to improve its reviewer recognition in recent years, but more work needs to be done. Additional avenues for recognition could come through Publons (as described previously), or through enhanced integration with Open Researcher and Contributor ID (ORCID).¹⁸ Authors, reviewers, and editors already can use ORCID authentication to log into the Journal’s manuscript management system, and additional functionality is being piloted by some journals to facilitate recognition and exchange of information about reviewing activities.¹⁹ No matter the chosen means, it is imperative that journals enhance recognition of reviewers in a substantive and sustainable manner.

As also noted by Haines and colleagues, no widely used standardized method of evaluating the quality of manuscripts or peer reviews is available.¹⁴ Significant research into the metrics of quality for peer reviews (that would help comprise a rubric) also is lacking. One instrument published by van Rooyen and colleagues in 1999 has shown promise, including good psychometric data, but is organized around whether certain content is present in the review rather than elements of quality.²⁰ With a standardized methodology for evaluation of peer reviews implemented, journals could correspondingly increase quality assurance among members of the editorial team by employing regular psychometric analyses of editors’ peer review evaluations.

Under the Journal’s current system, which is common across many scholarly publications, peer reviews are rated by an editor on a 1-100 scale. This score then becomes part of the reviewer’s composite rating within the manuscript management system. However, no set standards exist for the components of this 1-100 scale. Other than receiving a copy of the final editorial decision and a copy of the other reviews for a particular manuscript (which are positive recent improvements related to feedback), reviewers do not often receive direct meaningful feedback regarding the quality of their work. The previously mentioned study from Wiley identified the desire to receive feedback from a journal as the top reward that would make respondents more likely to accept an invitation to peer review.¹⁷ A standardized evaluation methodology (as discussed earlier) could assist editors in providing this feedback in a systematic manner that would allow for improvement.

The Journal has convened a task force to address the challenge of improving the peer review process as it relates to our publication, but the solutions are not easy. The current system is familiar and common and has sustained robust growth and changes to both practice and education over many years. Complete remodeling and restructuring of the peer review system would be ill-advised. However, as currently structured, reviewers receive limited feedback from editors on the overall quality of their reviews and even less feedback on how they can improve. From an academic standpoint, the information asymmetry, misaligned incentives, and lack of transparency of the process would seem to be unsustainable. Just as feedback on a publication helps the overall product improve, feedback on peer reviews would allow for individuals to improve as well.

• Received May 1, 2018.
• Accepted May 7, 2018.

• © 2018 American Association of Colleges of Pharmacy