Grant giving relies heavily on peer review for the assessment of the quality of proposals but the evidence of effects of these procedures is scarce

Demicheli V, Di Pietrantonj C

Published Online:

April 16, 2008

This review was carried out in order to assess the effect of the various processes of peer review on the quality of funded research. Only ten studies were included and described in the review. We were unable to find comparative studies assessing the actual effect of peer review procedures on the quality of the funded researchThere is little empirical evidence on the effects of grant giving peer review. Experimental studies assessing the effects of grant giving peer-review on importance, relevance, usefulness, soundness of methods, soundness of ethics, completeness and accuracy of funded research are urgently needed. Practices aimed to control and evaluate the potentially negative effects of peer review should be implemented meanwhile.

Abstract (click to read)

Background:

Grant giving relies heavily on peer review for the assessment of the quality of proposals but the evidence of effects of these procedures is scarce.

Objectives:

To estimate the effect of grant giving peer review processes on importance, relevance, usefulness, soundness of methods, soundness of ethics, completeness and accuracy of funded research.

Search strategy:

Electronic database searches and citation searches; researchers in the field were contacted.

Selection criteria:

Prospective or retrospective comparative studies with two or more comparison groups assessing different interventions or one intervention against doing nothing. Interventions may regard different ways of screening, assigning or masking submissions, different ways of eliciting opinions or different decision making procedures. Only original research proposals and quality outcome measures were considered.

Data collection and analysis:

Studies were read, classified and described according to their design and study question. No quantitative analysis was performed.

Main results:

Ten studies were included. Two studies assessed the effect of different ways of screening submissions, one study compared open versus blinded peer review and three studies assessed the effect of different decision making procedures. Four studies considered agreement of the results of peer review processes as the outcome measure. Screening procedures appear to have little effect on the result of the peer review process. Open peer reviewers behave differently from blinded ones. Studies on decision-making procedures gave conflicting results. Agreement among reviewers and between different ways of assigning proposals or eliciting opinions was usually high.

Authors' conclusions:

There is little empirical evidence on the effects of grant giving peer review. No studies assessing the impact of peer review on the quality of funded research are presently available. Experimental studies assessing the effects of grant giving peer review on importance, relevance, usefulness, soundness of methods, soundness of ethics, completeness and accuracy of funded research are urgently needed. Practices aimed to control and evaluate the potentially negative effects of peer review should be implemented meanwhile.

This record should be cited as:

Demicheli V, Di Pietrantonj C. Peer review for improving the quality of grant applications. Cochrane Database of Systematic Reviews 2007, Issue 2. Art. No.: MR000003. DOI: 10.1002/14651858.MR000003.pub2

Assessed as up to date:

February 20, 2007

Find the research

Get full text at The Cochrane Library

Primary Review Group:

Methodology Review Group

Health topics:

Methodological & diagnostic > Study methodology

Cochrane Summaries^beta

Independent high-quality evidence for health care decision making

Grant giving relies heavily on peer review for the assessment of the quality of proposals but the evidence of effects of these procedures is scarce

More like this

Copyright © 2012 - The Cochrane Collaboration