User:Daniel Mietchen/Notebook/Open Science/2011/03/02/2011 inquiry into peer review by the UK Parliament's Science and Technology Committee

The UK Parliament's Science and Technology Committee are currently running an inquiry into peer review (for both grants and manuscripts). Comments are invited from scientists whose material has been peer reviewed, those who commission peer reviews and those who carry out peer review. Deadline: March 10. Below are some of my notes on the matter. Feel free to chime in.

Strengths and weaknesses of peer review as a quality control mechanism for scientists, publishers and the public

Use table/ Google spreadsheet?

Clarification of terminology

• Treat manuscript and proposal reviews separately
• Distinguish between pre-publication and post-publication review (for manuscripts) as well as pre-funding and post-funding review for grants, and person-centric versus project-centric funding

Purposes of peer review

Both manuscripts and grant proposals

• Assessing methodological soundness
• Assessing proper contextualization with respect to prior research
• Impact guesstimation

Manuscripts

• Assessing validity of conclusions based on reported data
• Selecting a publication venue that is appropriate according to a perceived hierarchy of journals

Grant proposals

• Assessing appropriateness of budget and other resource use
• Deciding whether to fund a project, under what funding scheme, and under which conditions

Measures to strengthen peer review

• Make reviews public (possibly with an option for reviewers to remain anonymous)
• Allow interaction between reviewers and reviewees
• Reduce the length of proposals (cf. Gates Foundation) or papers
• Altmetrics

Value and use of peer reviewed science on advancing and testing scientific knowledge

• Post-publication peer review: Tremendously important; the essential ingredient of scientific progress
• Pre-publication peer review: much less so

Value and use of peer reviewed science in informing public debate

• Post-publication peer review: Highly valued, but so may be popular beliefs
• Pre-publication peer review: much less so, and only a small area in the medial landscape
• Pre-funding peer review: typically does not see public debate (exceptions: ITER and the like), but could benefit from it
• Post-funding peer review: typically does not see public debate (exceptions: ITER, Nobel & IgNobel and the like), but could benefit from it

Extent to which peer review varies between scientific disciplines and between countries across the world

• Any such effects can be reduced by bringing the process online and into the public sphere

Processes by which reviewers with the requisite skills and knowledge are identified, in particular as the volume of multi-disciplinary research increases

Assuming you want to have three reviewers per grant proposal, you could demand that submitters arrange for two of them to be posted in public along with the proposal, with the names of the reviewers being public. A third reviewer would be chosen by you to review the proposal and to comment on the two existing reviews. The reviews and comments by this third reviewer would be public as well but she could choose to remain anonymous to the public (she will always be known to you). In addition to that, you could invite comments from the submitters and the public, and add a karma system (think Stack Overflow) that allows to evaluate contributions to the system and aggregates these evaluations into some sort of reputation.

Such a scenario would have the following effects (amongst others), compared to the current system:

• The quality of submissions would be higher, as submitters would know that if they submit low-quality items, the whole relevant community could find out (this effect is known from journals which practice public peer review). Furthermore, the need to get two (public) reviews acts as an additional filter.
• The quality of reviews generally rises when they are to be published (again, this effect is known from journals). This effect is somewhat modulated by the reviewer's identity being known or not: If it is, reviewers tend to be more kind and less critical than if they can hide behind anonymity. Reviewer 3 as described above would balance that out.
• You would get an initial (though possibly somewhat biased) review along with each submission, and could then select reviewer 3 (or 4 even) such that she can cover those aspects of the grant that need further evaluation.
• You have to spend less work on finding reviewers and could divert some of your efforts into making the conversations around these proposals useful to the community, thereby increasing your pool of potential reviewers on the fly.

Impact of IT and greater use of online resources on the peer review process

• Essential

Mention some recent examples of blog-based post-publication peer review

Possible alternatives to peer review

Again, clarify terminology.

Further notes

• What would research funding look like if it were invented today?