Advocacy Archive
Restructuring of Behavioral Science Peer Review at NIH
June 3, 1998
Dr. Ellie Ehrenfeld, Director
Center for Scientific Review
National Institutes of Health
Rockledge Bldg.Two, Rm. 3100
6701 Rockledge Drive
Bethesda, MD 20892-7776
Dear Ellie:
We have been asked by participants of the 1998 Summit of Psychological Science Societies to convey to you their views on the restructuring of behavioral science peer review at the National Institutes of Health. First, though, we would like to provide a bit of background on the Summit to give you a sense of who is commenting.
The Summit of Psychological Science Societies is a periodically held event that convenes leaders from the many organizations representing psychological and behavioral science. The purpose is for scientists who represent many different behavioral research perspectives to come together to discuss common issues across the subdisciplines of psychology. Five Summits have taken place over the last decade. More than 150 people representing more than 90 organizations attended the most recent Summit which was held in Santa Barbara, CA, at the beginning of May. (The roster of Summit participants and organizations is attached FYI.) The general topic under discussion at the Summit was "Advancing the Science of Psychology."
During the Summit, a workgroup session was devoted specifically to discussing NIH peer review changes, and the topic was discussed again during plenary sessions. In the course of those discussions, the participants were well-briefed on the status of the reorganization by Virginia Cain from the NIH Office of Behavioral and Social Sciences Research and Ellen Stover from the National Institute of Mental Health. The consensus of the Summit is that behavioral peer review restructuring is currently the most important issue at NIH for us. Issues like the overall increase in NIH funding or priorities at various Institutes are secondary to whether there is an NIH peer review system that is sensitive to behavioral research. In this letter, we hope to convey the views of Summit participants that are in response to specific aspects of the peer review reorganization process as well as their views about peer review more generally.
Composition of Review Panels The single most important set of issues for Summit participants is whether there will be adequate representation of behavioral scientists on newly created review panels, and whether the review panels themselves will adequately cover the range of behavioral research that is fundable by NIH. The overriding concern here is the need for applications to be appropriately reviewed by those who are experts both in the methodologies and the knowledge base of behavioral science. You have no doubt heard similar concerns from other communities. However, the experience of the behavioral science community is that behavioral science is less well known than the biological sciences by NIH leaders, and is seen by those same leaders as less central to the NIH mission than the biological sciences. Still, the Summit participants recognized that the process now underway as described by Virginia and Ellen is explicitly designed to be sensitive to behavioral science research issues, and most involved in this process agree that NIH is making a strong effort to systematically consult with the behavioral science research community. Nonetheless, we want to underscore the importance given by Summit participants to ensuring adequate representation of behavioral science researchers and the range of behavioral science research once panels are formed.
Recruitment of Panel Members There was a sense expressed at the Summit, especially among senior researchers who previously served on NIH study sections, that service on peer review panels is becoming too onerous. The causes listed were many -- the workload, the length of term on the panel, the compressed time period for the review (someone suggested that a review conducted in 13-18 minutes is not a research review in any real sense, but more a presentation of an application by a primary reviewer followed by a vote), and the perception that sitting on a study section may interfere with the review of one=s own grant application. Yet, there was also a sense agreed to at the Summit that more senior researchers ought to be sitting on study sections -- particularly in the first rounds under a new peer review structure. Several suggestions for recruiting and retaining senior researchers on peer review panels were offered. We would be happy to expand on any of them if you like. They included reducing the length of terms, from four years to three or two years. An alternative was to allow panel members to miss a meeting or so per year over the course of a term. Other specifics were offered, but, generally, the sense of the group was to encourage NIH to be flexible in negotiating study section membership with potential reviewers.
Reducing a reviewer's workload was recognized as problematic by the group, but many encouraged NIH to look to NSF, DOD or even to Canada to see if techniques used there, such as mail reviews, might somehow be incorporated in the NIH system both to lighten a reviewer's load and to allow for more substantive discussion by a review group. The Summit recognized that NIH already has been trying to address these issues by allowing ad-hoc reviewers to vote on applications and by trying to initially screen out applications that have no chance at being funded. However, the experience of many at the Summit was that these attempts have not solved much. For example, a study section "culture" develops that does not easily incorporate an ad-hoc reviewer, even if the reviewer is allowed to vote on a particular application. Also, the screening out that has been done to date seems not to have affected more than a handful of applications per study section in any given round.
One of the more interesting suggestions came in response to the perception that study section membership puts one's own application at risk during a review. The Summit discussed whether, in fact, this perception is accurate. Study section members are sophisticated grant getters and we recall NIMH data from several years ago that the chances of a competitive renewal being approved while a grantee is on study section may be higher than for grantees who are not on study section. Given this, Summit participants mused over an NIH policy building on this information that would continue a grant for the period of time that one is on study section. That is, a reviewer's grant would not be able to be terminated while that reviewer was serving a term on a study section. This would send a dramatic signal to the community that NIH is serious about wanting the best researchers to serve, and it might not mean much financial cost to NIH. We use the term "mused" here because the Summit discussants appreciated the lightness of this suggestion. We raise it here as an example of encouraging NIH to think creatively about how to attract the best researchers to be peer reviewers. Another suggestion is that, since the new peer review structure is based on the assumption that there will be considerable overlap between study sections so any application ought to be able to be reviewed by more than one panel, a panel member's application should be directed toward another standing review committee, rather than a special committee (as is now commonly the case).
Evaluation The assessment of the peer review of behavioral science was a primary concern at the Summit. In its simplest form, the question is, will the restructured NIH system, including its incorporation of the review systems of the National Institute of Mental Health, the National Institute of Drug Abuse, and the National Institute of Alcohol Abuse and Alcoholism, work at least as well as the old system? Several points were brought out in this discussion. First, one way of looking at this change is that it parallels a Fortune 500 company changing its accounting systems. A standard approach to this kind of change in industry would be to run the new accounting system against the old one for some period of time to see if the results mesh. Ideally, an NIH evaluation should be conducted to compare the review of applications under the old systems with the review of applications under the new system. (Note that we are not saying that the results of research review under both systems should be identical.)
We appreciate that having the systems work in parallel for a time is not feasible. But maybe a quasi-experimental approach could be developed that would test features of parallel models. For example, NIH could re-review some applications in the new system that were reviewed under the old. A twist on this theme would be to take advantage of the built-in overlap between study sections and have some current applications reviewed in more than one study section. We can imagine a number of other techniques that might get at these issues. Similarly, there are a number of outcome measures that might be useful in these tests. One gross measure might be whether high impact researchers (NAS members, MERIT Awardees, etc.) get similarly good reviews in the different systems or from different study sections. Other, more fine-tuned measures could be developed.
This brings us to another evaluation issue that was raised at the Summit. The science of evaluation is itself a behavioral science discipline and NIH ought to be taking advantage of both its internal behavioral expertise and the expertise among the extramural behavioral science community as it considers evaluation strategies. Different methodologies, measures, controls, analyses, etc. need to be planned with the same kind of rigor as any NIH-funded research project. And it is behavioral scientists who ought to be involved in the design and implementation of an evaluation of the peer review reorganization. We would be pleased to provide a list of names of experts in this area if you would like.
Finally, the Summit participants recognized that the restructuring of peer review was just the immediate task, however important that is. Longer term questions of behavioral science peer review will continue even after the restructuring is complete. For example, how should NIH behavioral peer review change as new developments in the field emerge? We know that you have been thinking along these same lines and have been considering bringing a senior behavioral scientist to NIH in some capacity to advise on these very things. The Summit participants were delighted at this prospect and encourage you to go ahead. We would be pleased to help provide names of individuals to serve in this capacity or even to develop a group of behavioral scientists who might serve as advisors for you or for whoever you recruit for this effort.
Your consideration of these issues as NIH continues to restructure its behavioral science peer review system is greatly appreciated, as is your effort to consult with the research community in designing the new system. We look forward to continuing to work with you and would be pleased to discuss the issues raised in this letter as well as any others relating to behavioral science peer review at NIH.
Sincerely,
Alan G. Kraut, Executive Director, American Psychological Society
Richard McCarty, Executive Director for Science, American Psychological Association
cc: Virginia Cain, Ellen Stover, Norman Anderson