Shortcomings of the FSP Index

I’ve learned this morning,  from a comment to my last post and from an e-mail from a friend, about a problem with Academic Analytics’ Faculty Scholarly Productivity Index.  In putting together the data for the index, Academic Analytics used the database company, SCOPUS, which bills itself primarily as covering life science, health science, physical science and social science. I looked through their spreadsheet of journals and databases, and it did include the Philosophy Documentation Center, but this is not as much as one would hope for.  So I contacted Bill Savage at Academic Analytics and asked him about this.  He told me that they knew of the issue but thought it would be ameliorated because people in the humanities primarily publish books.  I told him that this was not at all the case in the dominant strand of philosophy in the States, analytic philosophy, though it was true of other strands (continental, critical race theory, feminism, critical theory).  So in effect, the FSPI, which gives significantly more weight to books than journal articles, does not accurately gauge the productivity of all philosophy programs in the U.S.  Savage said that SCOPUS is working on adding more and more journals to their database, so future rankings should be more accurate.

So, dear readers, the jury is still out.  I’m glad that the index, even with its weaknesses, shows the good work that under-recognized departments are doing.  Based on two and a half years of studying and applying survey research methodology (more than a decade ago), I still think the Philosophical Gourmet is a poor indicator of anything beyond what the people who are asked to respond to the survey happen to think. The findings are not generalizable. In other words, if you want to know how 270 people in high-brow departments gauge their colleagues, read the Leiter report.  But if you want a real gauge of what the profession as a whole thinks or of the quality of various institutions in the English speaking world, look elsewhere.

In the end, I think the best gauge of a graduate program is to be had by talking with search committees at the hundreds of colleges and universities who hire new faculty year after year.  In the end, it’s not how we rate the faculty as much as what kind of teachers and scholars emerge from a program.