Advocacy Archive
Redesign of NIH Peer Review: The “Boundaries� Report
submitted via email
American Psychological Society
1010 Vermont Avenue, NW - #1100
Washington, D.C. 20005-4907
October 15, 1999
Panel on Scientific Boundaries for Review
Center for Scientific Review
National Institutes of Health
Bethesda, MD 20892
Dear Colleagues:
We appreciate the opportunity to submit the following regarding your draft report, "Recommendations for Change at the NIH's Center for Scientific Review." These comments are provided on behalf of the 15,000 members of the American Psychological Society (APS), the majority of whom are academic researchers. Many hundreds of our members receive research and training support from the National Institutes of Health, and the status of NIH peer review is of central concern to us.
Your draft report is a thoughtful and well-intended attempt to grapple with the complex issues associated with scientific peer review. We concur in the objectives of creating a system that allows NIH to capitalize on advances in research and that anticipates future directions in science. And we agree that there is a need to shape the "culture" of the peer review system to ensure that the policies governing the structural design of the system are reflected at the operational level. These issues also were among our overriding concerns during the recent reorganization of the CSR peer review structure for behavioral and social sciences.
Now, speaking from the perspective of behavioral science research, we have a number of issues that we want to raise with regard to the draft report, ranging from its overall tone and scope, to concerns about specific recommendations. These issues are summarized below.
Scope of the Draft Report — We are concerned that the draft report appears to focus only on biomedical research. For example, the language of the report refers only to "the biomedical research enterprise" rather than "biomedical and behavioral" research, which is the more generally accepted phrase, and to medicine rather than health and health care more broadly. In contrast, you explicitly state in Appendix II, item 3, that the new IRGs created in the behavioral and social science reorganization have been incorporated into your IRG framework. In order to more accurately reflect the actual scope of the report and the significant behavioral research peer review component in CSR, we ask you to revise the language of the report to include behavioral research. I cannot emphasize enough that we are not simply talking semantics here or focusing on a small point. We are suggesting that the basic mind-set conveyed in the draft report is much more narrow than the actual NIH research agenda it purports to encompass. We are told that perhaps more than $1 billion of the NIH research budget is spent on behavior. That certainly shouldn't be ignored in the language of any general NIH report.
Impact on the Reorganization of Behavioral Science Peer Review — As noted in the previous paragraph, the report indicates that the proposed IRG framework incorporates the new behavioral and social science IRGs. However, it is not clear how the panel's new IRG design or its other recommendations will affect the new structure for the behavioral and social sciences research. In the absence of any discussion on these issues, the report raises concerns that the panel's proposed IRG structure might supercede aspects of that reorganization, or that the new behavioral and social science peer review might be altered in some way. Given the enormous effort by NIH and the scientific community to develop the new review structure, and also given the fact that the panel is drawing on that experience in designing the new peer review system, we feel it is in the best interest of NIH and of the field to maintain the new structure and to evaluate the impact of the changes (an issue discussed in greater detail below). We ask the panel to address these issues in the next version of its report.
Absence of a Basic Behavioral Research IRG — The first five IRGs in the panel's proposed system are focused on basic research into the fundamental processes underlying health and disease. We are alarmed at the absence of an IRG for basic behavioral science research, given the substantial basic behavioral science research portfolio at NIH and the fundamental behavioral questions associated with so many health issues. The lack of an IRG in basic behavioral science also reinforces the erroneous perception that behavioral science involves only applied research. Again, we see this as an NIH mind-set that functionally ignores what NIH funds in behavioral science. We ask the panel to address this issue and to include a basic behavioral IRG in its design.
Assessing the Impact of Peer Review Feedback in the Redesign of Research Proposals -- Appendix II, item 5, of the draft report refers to a report from the CSR liaison to the behavioral and social sciences research community. Specifically, the panel attributes to this liaison the claim that "behavioral and social scientists tend to overemphasize methodological deficiencies in their reviews of grants, with the result that study sections rather than investigators are often designing research programs in these areas." This paragraph is troubling for a number of reasons. Citing the perspective of one individual, even if that individual purports to be reflecting the community, is not a satisfactory basis for making policy. There should be a systematic attempt to confirm whether this occurs uniquely in behavioral and social sciences, or whether, as we suspect, it is merely a myth that is perpetuated in the absence of data. For example, items 6 and 7 under "cultural norms" give the clear impression that redesigning by reviewers is a concern across the peer review system as a whole and is not limited to behavioral and social science research proposals. In addition, we understand that the NIH Office of Behavioral and Social Sciences Research has begun to collect data on this issue and its preliminary results suggest that this general phenomenon of behavioral scientists being too critical of their own does not exist. We ask the panel to reconsider the statement in Appendix II, item 5, concerning behavioral and social science research and if it cannot be confirmed that the phenomenon in question is unique to these fields, that this be eliminated from the report.
Evaluation of Peer Review Changes — We are encouraged that the panel intends to follow the process used in the creation of new study sections in behavioral science and in neuroscience. We believe that process led to the creation of a much improved structure. We also are encouraged by the panel's strong statement calling for ongoing reassessment of the peer review system. We strongly advocated for a scientific evaluation of the effects of the reorganization of behavioral and social science peer review, and believe that such an evaluation is essential for the redesigned NIH peer review system as well. You may know that such an evaluation committee for the redesigned behavioral science review is now in place.
In its simplest form, the question is, Will the restructured NIH system work at least as well as the old system? Ideally, an NIH evaluation should be conducted to compare the review of applications under the old systems with the review of applications under the new system. (Note that we are not saying that the results of research review under both systems should be identical.) We appreciate that having the systems work in parallel for a time is not feasible. But a quasi-experimental approach could be developed that would test features of a parallel model. For example, NIH could re-review some proposals in the new system that were already reviewed under the old. A twist on this theme is to take advantage of the built-in overlap between study sections and have some current proposals reviewed in more than one study section. We can imagine a number of other techniques that might get at some these issues. Similarly, there are a number of outcome measures that might be useful in these tests. One gross measure might be whether high impact researchers (NAS members, MERIT Awardees, etc.) get similarly good reviews in the different systems or from different study sections. Other, more fine-tuned measures could be developed. Our point is that a science-based evaluation does need to occur.
Finally, we would like to point out that the science of evaluation is itself a behavioral science discipline and NIH ought to be taking advantage of both its internal behavioral expertise and that expertise present among the extramural community as it considers evaluation strategies. Different methodologies, measures, controls, analyses, etc. need to be planned with the same kind of rigor as any NIH-funded research project. And it is behavioral scientists who ought to be consulted for the design and implementation of an evaluation of the peer review reorganization. We would be pleased to provide names of experts in this area.
Thank you for the opportunity to provide these comments. Please don't hesitate to contact me if I can provide additional information on any of the issues raised, or to assist in your deliberations in any way.
Sincerely,
Alan G. Kraut
Executive Director
American Psychological Society