User:Daniel Mietchen/Notebook/Open Science/2011/03/02/2011 inquiry into peer review by the UK Parliament's Science and Technology Committee
The UK Parliament's Science and Technology Committee are currently running an inquiry into peer review (for both grants and manuscripts). Comments are invited from scientists whose material has been peer reviewed, those who commission peer reviews and those who carry out peer review. Deadline: March 10. Below are some of my notes on the matter. Feel free to chime in.
Strengths and weaknesses of peer review as a quality control mechanism for scientists, publishers and the public
Use table/ Google spreadsheet?
Clarification of terminology
- Treat manuscript and proposal reviews separately
- Distinguish between pre-publication and post-publication (for manuscripts) as well as pre-funding and post-funding for grants
Measures to strengthen peer review
- Make reviews public (possibly with an option for reviewers to remain anonymous)
Value and use of peer reviewed science on advancing and testing scientific knowledge
Value and use of peer reviewed science in informing public debate
Extent to which peer review varies between scientific disciplines and between countries across the world
Processes by which reviewers with the requisite skills and knowledge are identified, in particular as the volume of multi-disciplinary research increases
Impact of IT and greater use of online resources on the peer review process
Possible alternatives to peer review
Again, clarify terminology.
Further notes
Assuming you want to have three reviewers per grant proposal, you could demand that submitters arrange for two of them to be posted in public along with the proposal, with the names of the reviewers being public. A third reviewer would be chosen by you to review the proposal and to comment on the two existing reviews. The reviews and comments by this third reviewer would be public as well but she could choose to remain anonymous to the public (she will always be known to you). In addition to that, you could invite comments from the submitters and the public, and add a karma system (think Stack Overflow) that allows to evaluate contributions to the system and aggregates these evaluations into some sort of reputation.
Such a scenario would have the following effects (amongst others), compared to your current system: - The quality of submissions would be higher, as submitters would know that if they submit bullshit, the whole relevant community could find out (this effect is known from journals which practice public peer review). Furthermore, the need to get two (public) reviews acts as an additional filter. - The quality of reviews generally rises when they are to be published (again, this effect is known from journals). This effect is somewhat modulated by the reviewer's identity being known or not: If it is, reviewers tend to be more kind and less critical than if they can hide behind anonymity. Reviewer 3 as described above would balance that out. - You would get an initial (though possibly somewhat biased) review along with each submission, and could then select reviewer 3 (or 4 even) such that she can cover those aspects of the grant that need further evaluation. - You have to spend less work on finding reviewers and could divert some of your efforts into making the conversations around these proposals useful to the community, thereby increasing your pool of potential reviewers on the fly.
