Quick: Name the top five psychology departments in the world. Is that too hard? Well, what about North America? Is that any easier? What are five top departments in industrial/organizational psychology? In developmental psychology? In social psychology?

Americans love lists of ratings of practically anything. However, when we come to the upper reaches of academia, valid rankings of departments and programs are hard to come by. In fact, there are none.

Of course, there are rankings of psychology departments, just not any that most of us would consider methodologically sound. Several offerings appear on the home page of the Psychology Department at the University of California at San Diego (http://psy.ucsd.edu). The first listing of psychology departments is a “Faculty Citation Impact Ranking” in which — guess what? — UCSD ranks No.1. I rather like that ranking, because my department came in second. However, the UCSD Web site provides other rankings using other criteria with different results.

Interestingly, most of the rankings are based on a single criterion, such as impact as measured by citations (the first list on the UCSD homepage) and often the rankings, and the data on which they are based, are quite old. The National Research Council rankings, often still used for graduate programs, were published in 1992. Another NRC ranking is said to be in the works, but has been repeatedly delayed. The US News & World Report does rankings for graduate programs, including psychology and several specialties within psychology, but these rankings are simply based on department chairs’ ratings in a survey. The response rates in the surveys are not particularly high, and some of the results don’t pass the laugh test (or, more formally, have no face validity). Can a cognitive psychologist, one who is chair of a department without a clinical program, rank clinical psychology programs? I was chair at the time of the first US News ranking, and my opinion was asked of many departments of which I had little knowledge. I didn’t want to simply rank 90 percent of the departments with a “don’t know” so I hazarded guesses and suspect most of the (relatively few) other chairs who filled out the ratings did likewise.

When US News announced that they would begin ranking graduate programs via surveys of chairs in the late 1990s, it created consternation at a memorable meeting of the Council of Graduate Departments of Psychology (COGDOP). Many chairpersons spoke in favor of boycotting the entire procedure. Because the ratings would be entirely reputational, with no pretense towards collection of objective data, which is at least a part of the US News rankings of undergraduate programs, why should we chairs dignify the proceedings to help US News sell magazines? That’s a good question and one still worth asking. However, during the debate, I argued that we chairs should participate in the process. After all, US News was going to create its rankings, anyway, so we might as well cooperate and try to make them as good as possible. Looking at the matter with ridiculous optimism, perhaps we psychologists could even work with US News to make the ratings better. After all, psychologists take it upon themselves to measure almost every human quality — intelligence, creativity, need for achievement, and a hundred more (I can measure 10 or more varieties of memory myself). So can’t we devise a way to measure the graduate departments and programs of our own profession? Aren’t we rather fainthearted if we say to US News, “You can’t measure us and our graduate programs. We are too precious and complex to be captured in numbers.”

The COGDOP Board decided that year neither to endorse nor boycott the US News survey, but the rate of response among psychology chairs was rather low, less than 25 percent if memory serves. Nonetheless, university administrators, prospective graduate students and probably many other groups take the US News ratings of graduate programs seriously, too seriously given the flawed process used in arriving at them.

Now that I am no longer on the Board of the American Psychological Society, I can declare that it is high time APS did something to rectify this situation. Psychometricians are experts at devising measures of various qualities, and they know how to devise scales with separate components and weight them appropriately to arrive at a composite rating. Why not appoint a committee to create a ranking system for scientific departments of psychology? What better group to initiate this process than the Board of APS?

Embarking on such an enterprise of ranking departments is not for the timid. Rankings are, in some ways, a dangerous idea and one that is sure to bring howls of protest from egalitarians in our midst. “Aren’t all programs good in some ways?” I can hear people saying, in paraphrase of the dodo bird in Alice in Wonderland: “EVERYBODY has won, and all must have prizes.” The Board (and/or its specially appointed committee) would need to consider many issues. Through my finely honed precognitive powers, I can even anticipate the conversations the committee would have. I present a sampling of their deliberations below:

Psychologist A: “I am in favor of APS creating a committee to rank psychology departments. Of course, the devil is in the details, but let’s first see if there is agreement on the need to produce a ranking. Because psychologists are especially experienced in measurement of human qualities, we should lead the way. If we create a good means of ranking departments, then perhaps other academic disciplines will follow our lead.”

Psychologist B: “I am, of course, completely opposed to this idea. Graduate programs and graduate education have too many complexities. It would be foolhardy to try to do the type of ranking US News does for undergraduate programs, weighting various features, summing them, and fitting them on a 100-point scale. Then they use the scale to rank universities, so one program winds up as No. 6 because it received a 94 and another as tied for 15th because it received a 91, and so on. Psychologists know enough about measurement to know when it does not make sense to try to put complex, multidimensional programs on a simple linear scale.”

Psychologist C: “Good point, but I think rankings are inevitable — we have already seen that — and so we should strive to do them better. They won’t be perfect, but most anything would be better than what US News does for graduate programs now. We have a very low bar to leap over. However, if we do decide to rate programs, let’s get rid of the rankings of 1 through, say, 300. We would do better to take a tip from how restaurants and hotels are rated, with stars. It makes no sense to rank all the restaurants in the country on a linear scale of 1 to 500,000. Instead, some of the various guides rank restaurants as being “five star” restaurants, representing the most excellent. Four star restaurants are very good, and so on. The people who make the guides presumably factor in many qualities — menu, quality of the food, presentation, ambience, wine list, and so on. We could do the same thing with components that make sense for graduate programs. The Michelin Guide and others essentially have a five-point rating scale and, though there are faults with such a system, it is good enough. Are there really more than five gradations of graduate programs? By the way, Psychologist D, I can’t help but notice that you have been scribbling away the whole time that B and I have been talking.”

Psychologist D: “Yes, I have, but I was listening. I am glad to have the floor now, because I want to say that I agree with my good friend A that we should rank graduate programs. Whether we do it from 1 to 300 (the approximate number of PhD- granting programs in the United States) or by number of stars on a five-point scale is something we can decide later. But, as A said, the devil is in the details. What I have been scribbling, as you put it, is a list of the features or components that might go into our ratings. For the sake of argument, I would propose that we use the following factors: Reputational rank of the program as assessed by chairs (20 percent); number of tenured or tenure-track faculty (10 percent); total number of refereed journal articles published by the faculty over the past five years (20 percent); total number of citations accruing to faculty publications, counting books and chapters (20 percent); number of PhDs granted in the past five years who hold a job in scientific psychology, counting post-docs, faculty positions, research institutes, etc. (10 percent); selectivity of the graduate program as measured by the proportion of applicants who are admitted (10 percent); and finally, the yield of the program, the proportion of those admitted who attend (10 percent). I realize some of you might quibble with the components of these ratings, but we have to start somewhere.”

Psychologist E: “Quibble? Quibble? No, I strenuously object. You, D, are from the University of Michigan. You have rigged the components of the ratings to favor size over quality, because you have something like 150 faculty in your department. We must look at excellence of the average faculty member, not the total mass of faculty who might be milling about the place. I would take out the total number of faculty from your rating scheme. Rice University might be much smaller than Michigan, but that doesn’t mean it could not have a really excellent graduate program in certain fields. I don’t disagree with your other components, but instead of total publications and total citations, we should use mean citations and mean publications. That would be fairer and not favor the large programs. If you drop the number of faculty members from the calculation, I would add the 10 percent weighting to faculty citations. Citations are a direct reflection of the impact of one’s work, of the eminence of the faculty as evidenced by the attention paid to their contributions. Make citations 30 percent of the total weighting, and probably it should be more like half.”

Psychologist A: “Good points, E, but I see you accept the basic idea of ranking departments, so now we are quibbling over the details. But D, why do you include such a high percentage for reputational rank? Shouldn’t we use objective data and not a mere beauty contest? Let’s get rid of chairs’ rankings altogether.”

Psychologist E: “Look, A, I’m sorry to point out the obvious, but in rankings of departments, opinions of the leaders do matter. If all the chairs in the country believe that the Western Missouri College of Mines and Osteopathic Medicine has the best psychology department, then it does. Just like restaurants, it’s opinions that matter. I might have underweighted reputation at 20 percent.”

Psychologist A: “Come, come. Think of all the studies from social and cognitive psychology that show various illusions of judgment, ones in which people’s opinions do not track objective data, especially as the data change over time. I’m old, so I remember the Roose and Anderson reputational rankings of psychology departments in 1970. The California Institute of Technology received a high rating in that assessment, in the top 15. That sounds reasonable, until you stop to consider that Cal Tech did not then (and does not now) have a psychology department! Sure, there were a couple of great psychological scientists there, but the rankings were of departments and so Cal Tech was, as we say in the recognition memory business, a lure item. By definition, it deserved a zero. Similarly, there is the story of ratings of law schools where Princeton was rated something like 11th. But it’s the same story: Princeton doesn’t have a law school. Only by using more objective measures can we have truly worthwhile ratings that do not favor schools that have a famous name, as in the cases of Cal Tech and Princeton. If you had chairs rank clinical psychology programs, I bet Stanford would come out in the top 10 despite not having one. And there are other cases in which departments once had an excellent program that has now been surpassed by others, and yet we still see them at the top of reputational ranks. Opinions change very slowly because programs can ride on past reputations. We need to get rid of the reputational rankings that will only add halo effects to the data.”

Psychologist F: “Look, I’ve listened to this debate for as long as I can. Are you people living in the 1960s? Maybe back in the old days it made sense to rate departments. I’m not sure, since unlike some of you, I was not yet born. However, I can assure you that rankings of whole departments make absolutely no sense in 2005. No student really applies to graduate school to “a department.” Instead, students apply to a program within a department, and often they really apply to work with one particular professor. Almost all psychology departments, except at the very largest state schools, have made choices to specialize in certain areas of psychology and not in others. A university may have an outstanding program in social psychology and cognitive psychology, but no program in clinical psychology or I/O psychology. Or conversely, how do you rank these departments relative to one another? The answer is you don’t; instead, you rank programs in the different areas of psychology. That will be hard enough. For example, what do you do with an area like cognitive neuroscience, which often spans departments and even schools within a university, such as departments in a medical school and departments in arts and sciences?”

Psychologist B: “Good points, F. But let me muse along with you. Why stop at programs? After all, a social psychology program might be good in attitude change and in stereotyping, but not in other areas. A cognitive program might be strong in the study of memory, but weak in study of attention. Why not rate subprograms? Also, what about programs that look good on paper but only on paper. That is, there is a great set of faculty but they never speak to one another except to bicker, so the whole program is dysfunctional.

Psychologist G: “I realize we have many complications already, but I want to add another one. Are we really talking about rating all psychology programs? I recall from going to COGDOP meetings that there are over 600 graduate programs in psychology, but about half of those are masters programs. Are we rating those? And what about free-standing doctoral programs in clinical psychology? A, this ratings thing was your idea, so what do you think?”

Psychologist A: “Well, G, I think this one is easier than you make out. I said we should rate scientific programs in psychology. I don’t see why we couldn’t rate scientifi- cally oriented masters programs. Of course, many of the masters programs are applied programs and they might be fine, but we can let some other group rate them. As for the free-standing schools of psychology, we can let APA rate those. They accredit them.

Psychologist H: “The discussion so far has been quite parochial. We have just been talking about ranking American departments, but that’s very short-sighted for a Society that keeps saying it wants to have an international presence. What about all the excellent Canadian departments? And there are many all around the world: Israel, the United Kingdom, Germany, Australia, New Zealand, France, the Scandinavian countries and on and on. We really need a world-wide ranking, don’t we?

Psychologist C: That reminds me. The Brits do have a ranking system now, and I believe it’s on a five-point scale. I recall someone telling me that they even rank the publication outlets of the faculty for each paper. The US journals are weighted more favorably than the British journals, which is why we get so many submissions from the UK. Perhaps we should study their system and maybe we can adapt it for our own purposes?”

Alan Kraut: “I’m sorry to interrupt this discussion, but we have many more agenda items for our Board meeting, and we need to finish on time to go to the five star restaurant I have picked out for dinner. I propose that we call the question and take a series of votes on whether APS should rate scientific departments of psychology and, if so, what kind of ratings we should have. Would someone like to propose a motion?”

Unfortunately, just at this juncture my precognitive powers fail me. I don’t know what the APS Board will do. I end with an anecdote from my friends in philosophy. Ratings of philosophy departments are done by one person, a guy in Texas, and he publishes them every year as he watches how departments change. It sounds like a joke, I know, but the kicker is that everyone in philosophy reads the rankings and pays heed to them. If the APS Board doesn’t want to get into the rankings game, and if the National Research Council can never quite get its act together to provide another set of rankings, there is room for the enterprising scholar (preferably one with a knowledge of graduate education and psychometrics) to leap into the lurch.