Major research papers retracted over data controversy

Norman Swan: The extraordinary story of two COVID-19 papers which were retracted last week and have crystallised concerns about the quality of COVID-19 research, questioning what you can trust. Both studies were in prestigious medical journals and were based on analysing the experience of hospitalised patients sick with COVID-19. One was in the New England Journal of Medicine and claimed that two blood pressure medications which had been thought to make COVID-19 worse were in fact safe. The other paper was in the Lancet and showed that hydroxychloroquine in people with COVID-19 increased the chances of dying. That paper prompted the World Health Organisation to put a halt to hydroxychloroquine trials around the world.

The hospital data in both papers was provided by a tiny US company called Surgisphere, with no track record in the field and a tiny handful of employees of dubious background. An independent audit of the data had to be halted because they couldn't access information they had requested, and eventually last week the authors of both papers asked for the studies to be retracted. The saga which led to that point started days beforehand and was triggered by Dr James Watson and his colleagues at the Mahidol Oxford Tropical Medicine Research Institute in Thailand. I spoke to James just after both retractions.

James Watson: Hi Norman.

Norman Swan: How did you get onto this?

James Watson: So I work for a research unit in Thailand and we are running a very large prophylaxis trial of hydroxychloroquine or chloroquine versus placebo.

Norman Swan: And by 'prophylaxis' you mean you take it to see whether it prevents infection.

James Watson: Exactly, they will be health workers from hospitals all around the world, 40,000 in total, they will be taking it for three months and then we will be following them up and seeing whether they fall ill or not. The UK regulator had given approval ten days before, and then this Lancet paper came out, and a few hours after it was published, the investigators got an email saying 'you have to suspend immediately'.

Norman Swan: And this was before the Guardian story on the matter?

James Watson: Yes, this was long before the Guardian story, this was a good week before I'd say.

Norman Swan: And when he went to look at the paper, what did you conclude?

James Watson: We immediately saw that the statistical analysis wasn't very carefully done, and the big problem with these kind of observational data is that the patients who get the drug and the patients who don't get the drug, it's not a randomised allocation. Doctors in hospitals are giving drugs like hydroxychloroquine under this compassionate use, so that means patients who are very sick, it's kind of given as a last resort. And so when you compare the patients who have got the drug versus those who didn't get the drug, it's not exactly apples, it's kind of apples versus pears, so you have to make these adjustments, and the most important adjustment to make is for disease severity. And this hadn't been done very well in the analysis. So that was our first concern.

Norman Swan: And then when you looked further?

James Watson: Well, we were surprised that there were only four authors on the paper. We were very surprised that there were no acknowledgements to anybody else in the world. I mean, this is a global study, hundreds of hospitals around the world. We were also surprised that there wasn't really even a mention of which countries they were working in, it was just which continents. And then I sent an email to Andrew Gelman, who is a quite well-known statistician in Columbia University in America, and I sent him an email with some of my concerns, mostly statistical concerns. And then in the comments someone wrote; well actually it's interesting because there are more deaths in their dataset in Australia than were observed in hospital in Australia in that time. And that led us to then contact people we know in Australia, notably Professor Joshua Davis who is head of the Australasian Society for Infectious Diseases, he contacted the big hospitals in Australia because…I mean, I think there are only a few hospitals really that had a lot of coronavirus cases, and no one had heard of Surgisphere, no one had given them any data. And then he wrote a letter to the Lancet saying the data from Australia don't look real.

Norman Swan: And what about elsewhere? Did that trigger you to look elsewhere, because there are problems not just with Australia data.

James Watson: Yes, it did, and I'd say one of the major bits that looked a bit odd were the data from Africa. They had aggregated data from Africa, they didn't say which countries in Africa they had been working in, but if you tallied the numbers it would have meant they would have captured 40% of all the deaths that had occurred on the continent in that time period, and about 25% of all cases. And really the thing that didn't sound right was that this 40% of all deaths, all of these patients would have been under continuous ECG monitoring, which just doesn't really fit with the reality of most hospitals in South Africa.

Norman Swan: They've barely got ventilators.

James Watson: Exactly, exactly.

Norman Swan: So what did you do then?

James Watson: We were a bit annoyed by the fact that the authors of the paper replied to Professor Joshua Davis and said, you know, 'we are very sorry, a hospital in Asia got misclassified as a hospital in Australia'. And then we realised the Lancet was going to let them post a correction that said 'actually the numbers look funny but it's fine, it's just a misclassification problem'. And we sort of felt, well, the Lancet isn't taking this seriously enough because lots of things look weird, they really should be starting an investigation. So then we thought, well, we need to put pressure on the Lancet, so let's write an open letter, see who wants to sign it. And within about 12 hours of sending out some emails to our research network across the world, I had about 150 replies in my inbox very enthusiastically saying 'yes, this paper is dreadful, we want to sign up to this open letter'.

Norman Swan: Now, it wasn't just the Lancet, there was a paper in the New England Journal, which I mentioned before, which was into anti blood pressure drugs which were thought to make things worse but in fact this paper said it didn't make things worse, but based on the same dataset. Did you put the same pressure on the New England Journal?

James Watson: Yes, maybe a day after sending this open letter to the Lancet we…I mean, every day for the last two weeks we've been more and more worried about the dataset that these analyses are coming from. So then we thought, okay, well, let's dig into the New England Journal of Medicine paper, none of us had really read it carefully because it's not really in our area of research, but we looked at it carefully, and then we realised, well, numbers from the UK actually don't make any sense because the data from the New England Journal of Medicine paper were data collected very early on in the epidemic, so I think the last date was 15 March, so it was patients who had been admitted to hospital up until 15 March. They had about 700 patients from the UK in seven hospitals, but by 15 March, none of the hospitals in the UK would have had 100 patients. Actually we looked very carefully because most of the cases in that early part were in London, and actually none of the London boroughs had more than 100 positive tests, let alone 100 hospitalised patients.

And then there was also data in this paper from Turkey, three hospitals in Turkey, 350 patients. We contacted some people in Istanbul Medical University, and they said, well, actually our first patient we saw on 9 March, this is one of the biggest referral hospitals in Istanbul, and our second patient we saw on 16 March. So there's no way that there were 350 patients by 15 March, impossible. We posted our letter online, I also submitted it by the formal online system for submitting to the journal.

Norman Swan: So a couple of the authors wrote a letter to the Lancet to retract the paper, and then there's this extraordinary letter to the New England Journal of Medicine, all four authors, including Desai who runs Surgisphere, saying that this was a mistake and they didn't get access to the raw data, which is one of the most extraordinary…I know a little bit about scientific fraud, and two things jump out at me for this. One is that the people with the data got people with a reputation to be co-authors. The co-authors clearly didn't know enough about the study and just put their name to it. And now…I mean, this sounds like complete bullshit, they didn't have access to the raw data, when an independent auditor went in, I think prompted by the two authors that took the retraction in the Lancet, and they gave up on the independent audit because they couldn't get access to the data.

James Watson: Yes. I mean, I don't really know what to say, to be honest, it's so unusual, it's so odd.

Norman Swan: You must have got lots of information coming to you. What do we know about Surgisphere, and what could be the motivation for what at best is a monumental screwup but at worst could be scientific fraud?

James Watson: Well, quite a lot of journalists had been onto the Surgisphere case, and the thing that strikes one is that this company has no real history in data analytics, statistics, machine learning. They say they have the largest patient health record database in the world, but this is a tiny company…

Norman Swan: Three employees online or something like that.

James Watson: Yes, every report varies but none of them appear to have any knowledge about statistics, data analysis. One appears to be a model, another appears to be a science fiction writer. So you wonder how a company like this, so small, could set up contracts with hundreds of hospitals around the world, lots of different languages, lots of different legal systems, different ethical oversight systems. You wonder how that could happen, also in such a short period of time. They don't have any record of publishing on these very large patient databases before coronavirus.

Norman Swan: So the question then is who would have peer-reviewed this? Because presumably the New England Journal and the Lancet have a group of people and you send it out to people who are expert in this area, they would have sent it presumably to public health experts, to infectious disease experts and they would have sent it to data analytic experts. So it must have gone to two or three people for peer review in both journals. How did it get through?

James Watson: I guess the papers are quite well written from a clinical perspective. They do have reputable authors who are very good at what they do. I am surprised that the Lancet study did get through because I do think the stats aren't very well done, it's a bit sloppy. Obviously this is a problem with journals like the New England Journal of Medicine and the Lancet, is that the peer review process is not transparent.

Norman Swan: You don't know who got it, it's anonymous.

James Watson: Yes, you don't know who got it and even you don't know what they said. So the peer reviews could have been not very positive, we don't know. Other journals like the Public Library of Science, PLOS in America, some of the nature journals, they publish the peer reviews with the papers, and so you can see a critical outside view of the paper that is published alongside, which is very nice, but the Lancet and the New England Journal don't do that.

Norman Swan: What does this say moving forward? There's a lot of unpublished papers coming out, pre-peer-reviewed, what's called preprints. Whenever I report on one of those I tell the audience what it is and they take it with a pinch of salt. But we've got lots and lots of trials going on. As you say, you've got this massive trial going on. Just in your area there's a really lousy study just published in the New England Journal of a few hundred people recruited from the internet for a prevention trial, not proper testing afterwards, and it has been sung from the mountain tops and says it doesn't work for prevention. There's extraordinary things happening in this whole area. What is going on in this COVID-19 science area?

James Watson: There's this race to find treatments that work, to find the optimal strategies for dealing with Covid, and I guess because there's this urgency, things aren't done really as well as they should be. I do think some of these major journals have published quite a few underpowered studies, so by that I mean just studies that are too small to give a definitive effect. So very early on in Covid there was a study that got published on an antiretroviral, and the conclusion was, well, this doesn't work, but actually really the conclusion of that paper should have been this study was too small to know whether it works or not. And I agree with you, I think this prophylaxis study that was published yesterday in the New England Journal of Medicine, you could say, well, the main conclusion of that paper was not enough people were recruited and not enough people actually got infected in the control arm to actually understand and to know whether hydroxychloroquine protected those who were taking it as prophylaxis.

Norman Swan: Are we going to get honest data on vaccines, do you think, and other treatments?

James Watson: It's going to be very difficult to do studies in vaccines because of the nature of the epidemic. I mean, to test whether a vaccine works you need people who are exposed to the virus and who if they didn't get a vaccine would fall ill. We've had this exact problem with this prophylaxis study. When it was set up initially early February, everyone thought, well, the epidemic is going to kick off in Southeast Asia, so lots of science in Southeast Asia was put together. But then we realised, well actually for some reason it's not kicking off in Southeast Asia, so let's put some sites in Europe. And then there's lots of bureaucracy…these sites in Europe started a bit after this first wave. So now we are setting up sites in sub-Saharan Africa.

But you can imagine you're sort of chasing the epidemic around the world, but to get an answer you need to set up your site and do it at the right time in the right place, so it's very difficult. So vaccine studies will have exactly the same problems, and it could be very difficult to work out if they are effective or not. And you can imagine that you will have underpowered vaccine studies, and so a global vaccine might get rolled out that actually isn't very good.

Norman Swan: And let's hope there are not too many Surgispheres. Thanks for joining us on the Health Report.

James Watson: Thank you very much.

Norman Swan: Dr James Watson is a senior scientist at the Mahidol Oxford Tropical Medicine Research Institute in Thailand. We were unsuccessful in seeking an interview with the Lancet. Surgisphere posted a statement on their website in response to the retractions but it can be no longer accessed.