Nakamura Heads Peer Review at NIH

Chronology of a Career

1960-1963 Bronx High School of Science – has known since grade school that he will become a scientist.
1963-1968 Undergrad, Earlham College – ‘60s social and political climate piques interest in partying, and then psychology; eventually discovers experimental psychology.
1968-1969 Research Assistant, NY State – studies juvenile delinquency; develops deeper appreciation for how difficult it is to study human behavior, especially human antisocial behavior.
1969-1970 Research Assistant, Albert Einstein College of Medicine – pharmacology in rodents and primates.
1970-1976 Grad School – trains in research on primates and other animal models at NYU and SUNY Stony Brook (mentor is neuroscience pioneer Michael S. Gazzaniga); gets PhD in psychology from Stony Brook.
1976-1979 Post Doc at NIMH – conducts early cognitive neuroscience research with primates; is also supported by the National Eye Institute.
1979-1986 NIMH Intramural Staff Scientist – is enlisted in the effort to defend NIMH against charges from animal rights groups.
1986-1995 NIMH Extramural Program Officer – administers a biobehavioral extramural funding portfolio that becomes a Branch and is targeted by Congress; successfully defends animal and behavioral research.
1997-2007 Acting and then permanent NIMH Deputy Director – helps guide the agency’s mission; develops an increased appreciation of the interaction among behavior, brain function, and mental disorders. In year 2001, was acting Director of NIMH, for which he received the Presidential Rank Executive Merit Award in 2002.
2007-2011 Acting then permanent Scientific Director of NIMH Intramural Program overseeing approximately 900 staff in nearly 50 labs and branches.
2011-present Acting then permanent CSR Director – heads the peer review system for the National Institutes of Health.

Nakamura_WEB

Richard Nakamura

Nakamura began his career as one of the early researchers in what was to become the field of cognitive neuroscience. Knowing from an early age that he wanted to be a scientist, Nakamura took courses in the New School for Social Research and became fascinated by experimental psychology. He completed his BA in psychology at Earlham College and his graduate work in the primate research program of APS Past President Michael S. Gazzaniga at Albert Einstein College of Medicine.

His evolution from a bench scientist to a science policy leader took place against a backdrop of scientific and social change across several decades. Between undergraduate and graduate school, working on studies of juvenile delinquency, Nakamura first encountered the challenges in translating basic theoretical findings in behavioral science into effective interventions to address social problems such as recidivism among delinquents.

After obtaining his PhD in psychology from SUNY Stony Brook, Nakamura went to the National Institutes of Health (NIH) with a post-doctoral fellowship at NIMH. He then was supported by successive NRSAs from NIMH and the National Eye Institute. Nakamura soon joined the intramural program at NIMH where he studied the brain basis of visual behavior. Electrophysiological studies in the early 1980s formed the basis of theoretical papers on a possible role of cortical synchrony in behavior and decision making. He later switched to an NIMH extramural program position, and in the middle 1980s his program was at the center of congressional attacks on behavioral and animal models research. Nakamura’s success in developing the strategy for convincing legislators of the value of this research led to a full-time job as head of NIMH’s science policy office, and eventually to his appointment as Deputy Director of NIMH.

Now, as director of CSR, he leads CSR’s 450 scientists and administrative staff, overseeing their efforts to manage 80,000 incoming NIH grant applications a year and review the majority of them in CSR peer review groups. CSR holds 1,500 review meetings a year, involving about 16,000 reviewers from the scientific community.

 Alan Kraut: After many years at NIH, you’ve been named Director of the Center for Scientific Review, the place where peer review happens for most of NIH-funded research.

Richard Nakamura: Yes. We evaluate over 70% of applications coming to NIH. In general, CSR reviews the more basic science areas and the vast majority of R01s. The more complex clinically oriented mechanisms and institute-specific RFAs tend to be reviewed within the institutes. Between those poles, there’s a lot of discussion of who should do what. Remember that CSR provides merit scores for the applications we review, and the institutes take the scores and summary statements and make the award decision.

AK: What are some of the issues you’ve found facing NIH peer review? Are there changes you’re hoping to make at CSR?

RN: One issue has been to look at ways to maintain the efficiency of peer review. Taking all of the costs of running CSR and the number of applications we review, it costs less than $2,000 per application.

More importantly, we are paying a lot of attention to the efficacy of review: How good are we at top rating the grant applications that will have maximum payoff? There are those who say the only way to review applications is through review by outstanding scientists and that the main goal for CSR is engaging the best reviewers and focusing their efforts on the best applications. Another perspective is that there are objective measures out there — for instance, citation activity — and that these should be used in evaluating grant applications and the effectiveness of review. How can we effectively resolve such different perspectives? As scientists we should do research on these questions.

Bias in Peer Review

AK: It seems to me that one of the benefits of coming up in this system as a trained psychological scientist is that you have an appreciation for evaluation, statistics, and methodology that you can use in the kinds of organizational self-examinations that you’re talking about.

RN: That’s absolutely correct. In addition to the general problem of review efficacy, one of the biggest issues that I’m facing is whether our peer review is fair to all scientists. CSR’s reputation is based on a sense that we offer a fair playing field. Within that there are several questions: Is it fair across scientific areas? Are we properly evaluating the value of different kinds of science? Is it fair to sub-groupings of scientists? Is it fair to women? Is it fair to minority populations? We have found clear evidence that African Americans are receiving awards at much lower rates compared to scientists of other groups, including Hispanic scientists, White scientists, Asian scientists. This is very disturbing not only to me but to NIH generally, and Francis Collins, in particular. At his and his Advisory Council’s behest, we’ve looked at a number of different possibilities for an artifact like age differences, university differences, and so far we cannot come up with an easy explanation.

CSR has received permission to establish a research unit to really dig into these different issues, and we have set a priority on trying to determine if there is bias in our system, and if that bias could be based on race as opposed to something like quality of a scientific application. There are inherent biases in our system in that we always ask for reviewers with great scientific expertise. But this is a positive bias and we did not think that there could be a negative bias based on something like race. This has really activated NIH, and a key goal of my tenure will be to get the answer about what’s going on here.

AK: In fact, you met with some psychological scientists at the 2012 APS Convention to talk about possibilities, and I know that [APS Past President] John Cacioppo sits on your council.

RN: Yes, we’re expecting that psychological scientists will play a key role, helping us design experiments to determine if and how racial bias might be affecting our scoring system. Francis Collins has already appointed a subcommittee to his Advisory Committee to the Director that includes several psychologists to explore these issues. It reaches to the core of peer review, that we judge all scientists on an equal footing based on science quality alone.

Evaluating Changes in Peer Review

AK: Aside from the specifics of determining what may or may not be behind these apparent biased results, just the whole notion that you’re bringing science to bear on CSR procedures is an entirely new aspect.

RN: I wouldn’t say it’s entirely new. We’ve committed to doing more testing before we make changes and to establish a strong scientific base for peer review.

In addition to the issue of potential bias, we are looking to see if other forms of peer review might either be as good at a cheaper cost or better no matter what the cost in getting strong outcomes. For instance, there’s the possibility that editorial board types of review might do better than our standard face-to-face review. We are also comparing to see if video-based reviews can be as good as face-to-face reviews.

AK: And again, it’s got those same issues — what does “good” mean?

RN: Correct, developing measures of quality is essential. In general, we will do some of the things that we’ve done in the past — use questionnaires, gauge levels of satisfaction. But also we’ll be looking at whether various changes make a difference in which applications are selected? One possibility, for example, is to look at grants that were awarded after their applications were reviewed in one form of review versus another. Maybe 5 years out we could look at the resulting products, including publications, and see whether or not one set is more creative, more interesting, or has produced more compared to the others. Another possibility is to look at a measure like citation activity.

AK: Would any of this mean reviewing the same grants under two different systems? I mean, in some sense, you are the accounting system of a $30-billion NIH. And to compare systems or switch from one accounting system to another, most for-profit firms would run them in parallel for a little bit.

RN: Yes. We would like to take the same set of applications and run them through two different review processes. Because the results of such studies could easily be misinterpreted to the deficit of science as a whole, we want to be careful that there’s a lot of discussion about designs and measures and a lot of discussion on understanding what we expect to find from such kinds of studies.

It only requires a modest improvement over random choice for a peer review system to more than pay for itself, and we think that a modest improvement in that peer review system would more than pay for any increment in cost as well. But it would be really nice to have some data to show what’s improved.

We’ve essentially been doing the same form of peer review for a number of decades. We have tinkered with it a little bit. It’s very evolutionary. But other countries are doing different things, and here in the US the National Science Foundation does it differently. Understanding the pros and cons of each of these systems would be very valuable. Just because NIH has been so successful over the decades doesn’t mean we can’t do better, and it doesn’t mean another country or another agency might come up with a better system. No system of government, of manufacturing, or of human endeavor can afford to sit still. We’ve constantly got to re-think what we’re doing and figure out if we can do it better.

We’ve seen what happens when a country or an industry declares itself number one and thinks that it can rest on its laurels or just do evolutionary changes to stay on top. And we see that as the problem for NIH review as well. We need to keep moving, keep exploring, keep thinking, and be willing to take chances in order to keep NIH processes working well. Yet, we must avoid doing harm.

This is not something that’s going to be dreamed up out of my head or out of any subset of individuals here at NIH. This needs to be done as a scientific community. One of the goals here is to engage the extramural community not only in continuing the research they have done so well all along but to help us think about how to make peer review, how to make science review, a science itself. By all of us pulling together, we can emerge from this stronger than ever.

Replication in All Areas of Science

AK: Let’s talk a little bit about replication. It’s an initiative that we at APS are taking pretty seriously, as you know. But it seems to us that all scientists need to take replication seriously.

RN: NIH is very concerned about the replicability of its science. There are many cases in which attempts to replicate have shown that ability to replicate is pretty low and many interesting findings are not being replicated even though replications are being attempted.

AK: And you’re talking about science beyond psychological science?

RN: Absolutely. In some cases, as at the National Institute of Neurological Disorders and Stroke (NINDS), the director has created a fund to actively support replication studies, and they’re finding that the ability to replicate is low.

There is more interest in creating checklists of dos and don’ts of good science — the feeling is that there are too many shortcuts being taken or that some of the procedures that have long been accepted, like the 0.05% significance line, are inadequate to assure a scientific record which is valid.

A lot of self-examination is going on right now to try and make sure that those studies we are now paying for are in fact replicable and valid. But I think it’s up to all sectors of science to provide more rewards for scientists who do studies that ensure our literature is correct and valid and that all mentors and all students look at data and data collection with some skepticism to ensure that it’s being done well and done correctly. I am pleased to see that psychological scientists are so actively engaged in improving scientific methods.

Keeping the Enterprise Going

AK: I understand that the average age of an investigator getting their first funding of an NIH grant continues to go up. What about encouraging the next generation of scientists? What do you do here at CSR that would keep this scientific enterprise going?

RN: We’re all very concerned about the next generation. At CSR, we developed the Early Career Reviewer (ECR) program to help jumpstart the careers of emerging researchers by giving them an insider’s view of the NIH peer review process. They get tremendous insights into how to write a successful application and they get to network with distinguished scientists.

Of course, there is a problem here bigger than CSR. Right now, graduate students and post docs are seeing their mentors really struggling. It doesn’t matter what area, I hear clinicians fretting, translational scientists fretting, and basic scientists certainly worried. NIH has worked with the institutes to make sure that early career scientists are given breaks in receiving their initial grants.

We need to be very aware that other countries have increased their level of investment in science, in terms of the percentage of GDP, faster than the United States. A number of Asian countries have gone past the United States, and several European counties have now done so as well. These are 30- to 50-year investments, so the effect won’t be noticed right away, and by the time the effect is noticed, it will be too late. It will affect the next generation of Americans, and not just scientists.

You and I grew up in a time when United States’ investment in science was second to none, and we’ve seen the benefits of those investments. It has been fascinating to watch the growing knowledge of how science and human psychology and human behavior interact, that is our understanding of, not only the molecular basis, but the organ basis and the social basis, and how all of these different levels interact through a structure of the body. And we’re just scratching the surface. There’s so much to learn and so much science to apply to human behavior and so many ways behavior may change biology and health. It is frustrating to know that many of those problems are deeply addressable through the science that we all love.

Supporting Clinical Psychological Science

AK: Back in your NIMH days, back when you were deputy director under Steve Hyman, you were very encouraging of the newly forming clinical science movement in psychology, and you supported the development of a science-based training program in clinical psychology.

RN: Yes. It was clear that while scientific psychology was advancing rapidly, there were also strands within psychology that were not only not scientifically based, they were antiscientific.

It’s true that some scientific trends — for instance, the viewpoint that all biology was ultimately deterministic and that human behavior really was irrelevant once we understood the molecular mechanisms underlying it — proved to be incorrect. But as the science broadened, as psychology, and I would say neuroscience, started to come together to show that complex behaviors — such as social behavior, affection, love — and the neural basis of those could be visualized, one of the great revelations that has affected both our understanding of psychology and our understanding of where science will take us ultimately, has been the understanding that those emotions and behaviors can affect the capacity and the structure of the brain through direct effects on the biology of the genome.

AK: In fact, these days, to be a leading clinical psychologist, you have to know cognitive science, you have to know the biological basis of behavior, you have to know neuroscience, you have to know a fair amount of genetics.

RN: Yes. It’s been great to see the development of a science of a human behavior that spans many disciplines. I think psychological scientists are in one of the most exciting areas of discovery. Psychology has been one of the most open disciplines, and I just hope it’ll stay open and a leader in science.

Richard Nakamura on…

The Transition From Scientist to Science Funder
I had post-doctoral fellowships from both the National Institute of Mental Health (NIMH) and the National Eye Institute, doing cognitive neuroscience in monkeys, though at that time neuroscience had barely been named. Later, as a staff scientist at NIMH I helped defend NIMH against assertions of problems with research animals, but also came to embrace the changes to ensure that laboratory animals were consistently well treated with redundant protection systems.

For the next three years, I couldn’t make up my mind between science and administration. So I did both. On the extramural side I started working on a biobehavioral program, which ultimately became the Integrative Neuroscience Research Branch, while continuing to work on primate cognition, designing studies that others kept going, which kept me publishing papers through 2009.
Although I was reasonably successful as a scientist, I was better at helping other scientists do well in the field, so I decided to become a program officer to help others get NIMH funding.
Making the Case for Science
At one point, NIMH and particularly the portfolio of my branch came under attack by Congress. Some said we weren’t learning from animal research anything of use for understanding human mental illness, and the institute was threatened with a 30% budget cut. Naturally, I helped with the defense of the portfolio, and it turned out that this came to the attention of [ABC investigative reporter] Sam Donaldson, who did a lengthy broadcast piece on alleged examples of government waste. He said at the start of his report that he thought he had found “the mother lode of all government waste.” But after spending a significant amount of time talking to our scientists and digging into the issues in greater depth, he concluded that it was worth it at twice the price, that this really was very important work. That was critical in turning the issue around.

Understanding how the public, how scientists, how our patient advocacy communities see the utility of science, and understanding the congressional perspective about why they were funding science helped me make the case for science. This experience led to my appointment as head of policy for NIMH, and ultimately, to my appointment as deputy director.
The Example of Behavioral Science
I had been taught about the role of behavior and human interactions in changing the basis of behavior but initially didn’t believe it because I couldn’t understand the mechanism. I thought we could get to a deterministic understanding of the world, the molecular cause and effect for all of human behavior. But it has become clearer and clearer that nature has been ahead of us for a long time and humans have developed enormously complex ways to adapt.

Once, key research questions were, “How do we move from biological systems to understanding the brain, and from understanding the brain to human behavior? How do we integrate all of that kind of information?” Other important issues were linked to the question of the nature of mental illness: Was it fundamentally a biological problem? Was there a difference in the brains of individuals who have a mental illness? Was this a social phenomenon?

What we know now is that it is all the above. We now know that diseases and genetics can predispose an individual, and there may be changes in the brain before the onset of mental illness. And we also know that behavior and social interactions can fundamentally change the structure of the brain, and the genome; how DNA produces RNA and proteins can be determined by epigenetic factors controlled by the environment including social interactions.

This has all made psychological science more important, especially as it is informed by the richness of a tremendous amount of knowledge, of basic biology as well as what behavior and environment will do to change the mind and brain. I think this is a great time to be a scientist and a psychologist because the questions are so open. This is an incredibly exciting time, and yet enormously frustrating because funding isn’t sufficient.

Observer Vol.26, No.10 December, 2013

Leave a comment below and continue the conversation.

Comments

In my experience, one of the weakest component of the peer review system is the SRA, who is often superficially familiar with the research area she/he is trying to manage for review. Quite often, they follow bad advice in the selection of reviewers and end up with a mediocre group with inadequate judgement. Some SRAs act cynically and seem to practice favoritism. All of this is a complicated problem made worse by the tight budget.

Leave a comment.

Comments go live after a short delay. Thank you for contributing.

(required)

(required)