Abstract
<p>BackgroundDistributed peer review (DPR) of grant proposals, where applicants review the proposals of others applying to the same funding call, offers advantages, including increased feedback to applicants. With the reviewer pool being determined by the applicants to a particular call, there are, however, concerns that reviewers may not possess the necessary expertise and experience to provide quality reviews and feedback. We analysed review comments from recent trials of DPR for funding evaluation at the Volkswagen Foundation to obtain insights into reviewing and the impact of reviewer characteristics on review content. MethodReview comments across two years (N=2950) were classified (content and sentiment) using existing, fine-tuned machine learning classifier models. Mean prevalence of content categories and sentiment was calculated, both overall and stratified by reviewer/proposal characteristics. ResultsReview comments were consistent with review scores and overall ranking of proposals. Minor differences in content and sentiment according to reviewer gender and research experience (years working in research) were detected. There was minimal variation according to funding peer review experience and disciplinary alignment between reviewer and proposal. ConclusionExisting classifier models can be used to analyse grant review comments and to obtain insights into reviewer consistency and adherence to funding criteria. Concerns about the influence of reviewer experience and expertise on reviews may be unfounded. Results from analyses of review comments can be used to audit and refine review processes.</p>