A free service for scientific peer review and publishing

much ado about noting

A tale of two distributions

Janne-Tuomas Seppänen
25 Mar 2016

Imagine you are a scientist, and...

(well if you are reading this you probably are a scientist, but even if you're not, read on. You may learn something about us.)

... you are tasked by government to advice policy based on some data.

Your task: allocate very limited funds to most efficient uses in biodiversity conservation.

(if you're not a biologist, please hang on, this won't be about biology but will be about Nature)

Data 1 is a sample of 828 plots of land, varying in size from people's backyards to entire forests, and varying in type. You don't know if a given plot is a wheat field or a pristine forest or a parking lot, and you don't know how large it is. The only thing recorded is the count of individuals observed. And you don't have data on species either. They could have been butterflies, lizards, frogs, mammals, fungi. Furthermore, you don't know how many individuals the observers missed, just that they certainly missed the ones living underground, and those only active at night, and probably others too. The only numbers you have are the counts of individuals observed.

Data 2 is a sample of 615 plots of land. Here also you only have the counts of individuals observed. But in Data 2 all plots of land are in natural desert regions, and all are roughly the same size. You know the plots still differ in many unknown ways, but at least you know none of them are parking lots or wheat fields or tropical rainforests.

The plots of land are not a random sample drawn from the landscape. In both cases, the team who entered the plots into the set, chose some from a larger set of possible choices about once a week throughout 2014. The team based the choice on an a priori prediction (guided by two expert opinions) on which plots are likely to have lots of individuals. Teams for Data 1 and Data 2 are different, and did not interact in any way to coordinate their methods, nor did either record their methods. Observations naturally started only after a plot was chosen, so for some plots the number comes from over 700 days of observation, while for some it is from around 400 days of observation. The data is not normalized to be observations per day, oh no, it's the total count only.

Now, some propositions:

  1. Would you state that because the mean is higher, the team that chose plots for Data 1 should be tasked with choosing land areas for conservation?

  2. Would you state that because the mean is higher, any plot of land in Data 1 is more valuable for conservation than any plot of land in Data 2?

  3. Would you state that because the mean is higher, the best use of limited funds is to give government grants to landowners whose land is in Data 1 so they can get more plots of land to manage, over all landowners whose lands are in Data 2?

If you answer "yes" to any of those propositions, you should ignore the first five words of this post, and consider a career in some field with less stringent standards for truth. Like politics. Then you could be the one to give the assignment in the first place.

But of course you answered "no" to all of the above. You are a scientist, after all. For proposal 1., the distributions and the methods used do not come anywhere close to allowing simple comparisons of means. Proposal 2. is just silly. And proposal 3. is two steps beyond silly, firmly in the domain of crazy. To do any of these things would not only be stupid, but irresponsible, to the level where you should get attention from the state prosecutor if funds were wasted due to your statements.

At the very least, your career as a scientist would be destroyed. And that would be right and proper. The very limited funds available for science should be spent on people who at the very minimum understand distributions and can judge how methods and data limit analysis and interpretation.

Yeah, it's an allegory

In reality, Data 1 above is the citation distribution for articles that appeared in Nature in 2014, while Data 2 is the citation distribution for articles that appeared in Proceedings of the Royal Society B in 2014. "Individuals" are citations, and "species" are different kinds of citations from praise to method replication to criticism to tweets to everything else, and "plots" (=sampling units) are articles, and "size" is how broadly the article can be used, and "type" is the field of science of the article. The description above is an allegory for how article entities come to be and how citations are indexed, and I am afraid it is pretty accurate.

The mean I calculated roughly corresponds to journal's impact factor. But not exactly, which is not surprising given that the way it's officially calculated is more than a little opaque. Some even say it's negotiable between publisher and Thomson Reuters, but I don't know if the wrongness goes so far. But the misuse of that number is bad regardless of whether the calculation is malleable or not.

Right at this moment, thousands of senior scientists are doing proposition 3.

Even worse, they are doing it in grant committees. In tenure evaluation boards. In hiring decisions. They decide the fate of young scientists using the (non)logic of the propositions listed above.

Except they are not even looking at the distributions. They are using a single number. The impact factor.

When a young scientist is applying for a grant or a faculty position in science, the following unspoken rules usually hold:

  1. Editorial team in a journal with higher impact factor selects more important articles than other editorial teams. Their choices can be used to choose recipients for grants or positions.

  2. Any article in a journal with a higher impact factor is more important than any article in a journal with a lower impact factor.

  3. People who have articles in higher impact factor journals are better scientists than those with articles in lower impact factor journals.

Compare these propositions to the earlier list.

If you are a scientist, you already knew this happens, with feelings varying from indifference to despair to resignation. Though if you are in the age cohort of the guilty, you probably claim that it is only the other senior scientists doing that, not you. Which may be true; I know many responsible and wise senior scientists with utmost integrity, who are very concerned about this issue - maybe you're one of them.

If you are not a scientist, I hope you are a little bit angry. Because, dear citizen, this is how your taxes are allocated in the domain which is absolutely crucial for expanding humanity's knowledge, for bringing new technology, new cures, and solutions to our biggest problems.

There is hope, but it comes from an unexpected direction

Last week, I was at the Journal Summit, at the USA National Academy of Sciences. That same organisation publishes PNAS, which ranks fourth in impact factor for multidisciplinary science category, right after Nature, Science and Nature Communications. Surely NAS would not entertain a string of speakers attacking the stupidity of impact factor worship, to an audience that was mostly executives from publishing companies?

But they did. And based on reactions and my conversations afterwards, the audience mostly agreed.

NAS has a communication policy that prohibits live-tweeting or quoting anything said at the Summit without person's explicit permission. That worked well, as some of the exchanges between the floor and the podium were, well, very direct to put it mildly. You wouldn't hear such candid comments in settings where people with big responsibilities must constantly be aware that their words will get tweeted at any moment.

But I approached one speaker after his talk, to ask permission to quote some things he said. Because what he said was important, and because his position adds a certain weight and unexpected significance to what he said. I really think us scientists should hear, and consider, what the Publishing Director at the oldest and one of the most prestigious publishing organisations has to say about impact factor and about research assessment processes and behaviours.

I am very glad that he gave me that permission. He even sent me the text of the talk afterwards, so I can quote him verbatim.

Here are some selected quotes from what Stuart Taylor, Publishing Director at the Royal Society had to say:

There is a conflation of ‘quality’ with impact and a heavy reliance on citation based metrics (especially the Journal Impact Factor) in most research assessment processes.

...a research culture which largely evaluates scientists based on how many articles they have published in a handful of elite, prestige journals.

...a cycle of needlessly repeated peer review, publication delays, impact factor gaming, skewed research priorities and publication bias.

... the processes and behaviours which equate publishing in a top journal with being a successful scientist are deeply ingrained, as are those which regard the number of citations as the only worthwhile measure of quality.

Most of these problems, though, are not the fault of journals or their publishers and many are largely out of their control altogether. There are opportunities for publishers to exert influence and many have already made very beneficial changes. But they cannot do it alone.

...changing the system of research assessment to take a far broader view of a scientist’s achievements rather than merely which journal they publish their research in.

Perhaps above all, how their work is used by and influences the work of others.

We clearly can’t expect the early career researcher to take action alone - after all, they have the most to lose.

A concerted leadership effort will be needed from universities, funding bodies. learned societies and publishers to drive genuine and meaningful change in the evaluation and reward system.

Back to my comments, the following is not from Stuart's talk.

It is a shame that the academic community is the biggest obstacle for hope.

Dear colleagues in academia, many people in the publishing industry are way ahead of you on the path of fixing this. Which speaks volumes about them, but even more about us. But Stuart and other wise people in the publishing world can't fix things without you. So get on board. Signing DORA is a good place to start, but then you need to honor that declaration in your actions. I am proud to say Peerage of Science as an organisation is signatory number 132 (out of 623), and I've also signed it personally as the 445th signatory.

Especially if you are in a leadership position in academia, it is your duty to fix this, and to leave a better world for the next generation.

« | »

peerage of science Facebook peerage of science twitter peerage of science twitter