Crowd-Sourcing the Clones: A New Kind of Peer Review?
Editors of PubPeer were used to getting only about 400 unique visits a day to their post-publication peer-review website.
“Now we are getting that kind of traffic every 15 minutes,” said a PubPeer editor recently.
This astronomical leap began on May 22, when scientists posting anonymously on the site tweeted the news they had found errors—figure replications—in a recent globally hailed Cell paper. That landmark paper described the first successful cloning of human cells.1
As of 9 a.m. on May 24, about 48 hours after PubPeer tweeted its cloning error finds, those posts had been viewed a whopping 11,147 times. Almost a week later, the number of views is still rapidly climbing, up to 15,689 views—and beyond.2
While watching the count climb, young scientists anonymously staffing the novel site—formed in October to open peer-review of scientific papers to the scientific public—have been conducting interviews with “major news outlets around the world, answering emails, and responding to tweets,” the PubPeer editor says.
“The feedback has been extremely favorable, and has been very exciting for scientists used to sitting behind microscopes all day.”
The situation is eerily similar to that which occurred in 2004-2005, when South Korean scientist Woo Suk Hwang published two high-profile papers claiming to have cloned the first human cells. Anonymous commenters in that case, too, discovered figure replications, and reported them on an unaffiliated website (in that case, that of the Biological Research Information Center [BRIC]).
Those anonymous posts led to investigations finding that both Korean cloning papers were completely fabricated: not a single human cell had been cloned.3
This is highly unlikely to be the case here. The lead author on the Cell paper, Shoukhrat Mitalipov of the Oregon National Primate Research Center, spoke openly with Nature hours after his paper’s errors were observed. He said two of the replications involved simple mislabeling errors, and would be fixed. He added that they were minor. That day he also spoke to Science Insider, noting the cells were due to ship to third parties to be confirmed as clones as soon as he received institutional review board approval.
By contrast, the lead scientist on the 2004 and 2005 cloning papers, Seoul National University’s Woo Suk Hwang, had ducked all attempts to check his work for weeks, until his hand was forced by university and law officials. He was eventually indicted for embezzlement in what some call history’s largest science fraud (in terms of number of people involved).
Still, scientists in the stem cell field were not entirely surprised by the recent finding. Turnaround on the Cell paper had been very fast. The paper was submitted on April 30, and accepted for publication May 3. Usually that process takes months. Cell responded that the reviewers were all highly reputable, and had simply agreed to give the paper top priority.
Others were appalled by the déjà vu of it all.
Yet another major difference is, however, the anonymous young scientists behind the “catch” in the recent case. Pubpeer, unlike BRIC, is billing itself as a new—and it says necessary—phenomenon in science: a central clearinghouse for anonymous post-publication peer review. The revelations were carefully choreographed to give the paper’s authors time.
“`Peer 1’ put up the original post on Tuesday morning (May 21) and the authors were sent an email at that time,” says one of PubPeer’s editors. “We gave the authors 24 hours to react, and then tweeted the link to the comment. The story exploded on Twitter the next day, Wednesday, when (freelance science writer) Ed Yong retweeted it. Cell reacted via Twitter later that day (on Wednesday) when (University of California San Francisco stem cell researcher) Benoit Bruneau tweeted it to the Cell Twitter account. On Thursday, Cell made a statement on PubPeer and on the comment section of the original article.”
Please note, the editor added, “the efficiency of post-publication peer review in improving the quality of the article… We feel that a centralized database of post-pub review will be great for science if it achieves a critical mass of users.”
In recent months there has been some debate over the idea of a largely anonymous post-publication science website. PubPeer only allows as registered anonymous commenters the first and last authors on peer-reviewed papers listed on PubMed, a global registry of science publications. Those commenting on their own papers may comment under their own names. Posts by unregistered parties are pre- screened by PubPeer. One of its goals is the “democratization” of peer review. The anonymity was established to protect reviewers and editors.
But there are many potentially erroneous or indiscriminate snipers among that vast Pubmed crew, critics have said. Snipers in such situations can be overly focused on minutia. Anonymous web commentary, in general, has led to infamously unjust pile-ons, particularly in non-scientific areas.
Alan Trounson, head of the world’s largest stem cell funding body, the California Institute for Regenerative Medicine, commented by email: “There will always be these kinds of blog comments—made more popular by loose science reporting. I guess we recognize both their role and potential dangers. Makes it even more important that we try as scientists to be exact and accurate. I see trouble when these organizations have a political or religious agenda--maybe any pressure group agenda.”
Whole-hearted supporters of this kind of post-publication review point out that traditional peer-review involves plenty of injustices, as well. Journals send manuscripts for potential publication to a very small coterie of anonymous reviewers. Those teams can be far too small, and far too elite, contend PubPeer supporters. Politics can play an enormous role.
Says prominent National Institutes of Health immunologist Polly Matzinger, who prefers to sign her reviews for traditional journals: “The anonymity of pre-publication referees can lead to biased, nasty, and lazy reviews, yet anonymity is rampant in the "normal" reviewing process. We're all used to it, so it shouldn't come as a shock that there could be anonymity in a post-publication review. It's too new for most people to have thought about. But if we do start thinking about it (as I have now), then we should simply accept it.”
Overall, she contends, it might be best if everybody considers signing comments. “In some fields, such as math, I’ve been told that the referees are not always anonymous. Their names are listed on the paper and they take some responsibility. That would be difficult in biology, as it's hard to know how an experiment was done if you're not in the lab. However, I do think public reviews would help. There have been journals that published the reviews along with the papers. A great idea.”
She notes: “The kinds of mistakes in this stem cell paper should never have been allowed to be published. Science relies on basic human values of trust, tolerance, honesty, and respect. When any of these are lacking, scientific inquiry is lessened. We who do research cannot do everything ourselves. We must trust others to relay results honestly so we can build on them, so we don’t waste countless lives chasing mirages. Is it good to have post-publication review by readers of scientific papers? Of course! If reviews get too vituperative, they will be ignored. As long as the anonymous scientists doing the post-publication reviews are honest and useful, they will be heard and respected. At least, unlike the (also anonymous) referees used by the journals, their criticisms are public.”
Finally, many scientists are wondering: “Where were the traditional reviewers?”
“Why on earth didn’t the reviewers notice something as blatant as duplications of figures?” asks one scientist. “Did they not look at the data? Are data so irrelevant these days? So much data in many papers are now relegated to “supplemental” sections, which are usually in a separate place online. The reviewers could have done the authors a real service if they had done their jobs.”
One way or another, PubPeer now has astronomically higher visibility than it did mere days ago. That visibility may linger, for a while. Errors in two sets of the most high-profile papers in recent history had to be unearthed anonymously on the web. This is unlikely to be soon forgotten.
Still, traditional peer-review has been around a long time. Says a PubPeer editor: “Congratulate us when post-publication peer review becomes more commonplace.”
References
1. Tachibana M. et. al. “Human Embryonic Stem Cells Derived by Somatic Cell Nuclear Transfer,” Cell (May 15, 2024), epub ahead of print.
2. PubPeer: http://pubpeer.com/publications/F0CFE0360002C25DC0BEFE28987D70
3. Chong, S. et. al. “How Young Korean Researchers Helped Unearth a Scandal,” Science, Vol 311, Iss 5757 (January 6, 2024): p22-25
