Quality and completeness of medical literature questioned in two new studies
By Michael O’Riordan
New York, NY – Two recently published studies cast some unfavorable light on the current quality and completeness of medical literature, with one showing that less than half of registered studies are published in medical journals [1], and the other showing questionable discrepancies between the registered and reported clinical outcomes [2].
Dr Joseph Ross (Mount Sinai School of Medicine, New York), the lead investigator of the study showing that just 46% of studies registered on the National Institutes of Health (NIH)-funded website ClinicalTrials.gov ever make it as a published paper, told heartwire that the findings are very alarming.
“The research should send shockwaves through the research community, as it shows us that while it’s all well and good to practice evidence-based medicine, we don’t have all the evidence,” said Ross. “In terms of following guidelines and understanding the right treatment approach, we actually don’t have all the evidence at hand to make those decisions. This is a really shaky foundation. On top of that, the stuff that is being published might not even reflect the studies as they were designed.”
Dr Harlan Krumholz (Yale University School of Medicine, New Haven, CT), the senior author of the paper with Ross, commented to heartwire that the studies, in tandem, reveal significant shortcomings of the data available in the national registry of randomized trials, with the implication being that drug assessment is “hampered, even undermined” by the incomplete picture.
The study by Ross, Krumholz, and colleagues is published in the September 9, 2009 issue of PLoS Medicine, while the second study comparing the registered and published outcomes in clinical trials, which is led by Dr Sylvain Mathieu (Hôpital Bichat-Claude Bernard, Paris, France), is published in the September 2, 2009 issue of the Journal of the American Medical Association.
What’s required and optional with ClinicalTrials.gov?
In 2005, the International Committee of Medical Journal Editors (ICMJE) began a policy stating that information about a clinical trial needed to be registered before patient enrollment as a precondition for publication. ClinicalTrials.gov, a registry that has seen an average of 220 trials registered each week since 2005, requires mandatory information from investigators, with other information considered optional.
Mandatory reporting includes study title, summary, design, phase, type, conditions or focus of study, intervention, eligibility criteria, gender, minimum/maximum age, recruitment status, sponsor, facility, study official or facility contact, central contact, oversight authorities, and institutional review board approval, as well as other administrative details.
Optional information includes the primary purpose for the study, start date, completion data, enrollment target number, primary and secondary outcomes, whether the trial accepts healthy volunteers, and FDA product status.
Trial completed but never published
In the study by Ross et al, the researchers examined the reporting of registration information among a cross-section of clinical trials registered at ClinicalTrials.gov after December 31, 1999 and updated as completed by June 2007. The study looked at the trials registered and the completeness of the mandatory and optional reporting fields on the NIH website and the extent to which these registered trials were published.
Excluding phase 1 trials, the group identified 7515 completed trials over the seven-year period, and of these, nearly 100% reported information mandated by ClinicalTrials.gov, including the study intervention and sponsorship. Ross noted, however, that some trials were more complete than others, with some providing only vague or nondescript information. Optional data, however, fared worse. Just 53% of studies reported the study’s end date, only 66% reported the primary outcome, and 87% reported the trial start date.
“About half to two-thirds of the time, the information that was supposed to be there was there, or could have been there,” said Ross. “What we discovered as we looked through the second step at publication was this optional information is really important when you’re trying to figure out what’s been published and what hasn’t.”
To assess how many of the registered trials were published, the group examined 10% of the completed trials as a subsample. Of these trials, just 46% of studies were published, and of these, only 31% provided a citation in ClinicalTrials.gov of that publication. Trials sponsored by industry fared the worst, with just 40% of registered trials published, which was significantly less than the 56% of published trials not sponsored by government or industry, such as a university or foundation. Of studies sponsored by government, 47% were published.
“For the most part, everybody’s publication rates were disappointingly low,” said Ross. “It’s really hard to know why. On the one hand, it seems like zillions of papers are published every day, and it should be easy to get something published. But once a paper has been rejected a few times, it’s sort of easy to lose steam. Investigators and companies are often chasing the next grant for the next study by the time the push comes to get something published.”
Ross said the results are in line with literature showing that studies sponsored by industry are more likely to be positive. Because their publication rates are low, it is likely that many negative trials aren’t being published. He noted that it is also possible that they are conducting trials for purposes other than research, such as trying to get people to adhere to their medication, and while it is registered as a trial, it is not considered for publication.
Differing clinical end points
In the second study, by Mathieu and colleagues, the group compared the primary outcomes specified in ClinicalTrials.gov with the primary outcomes reported in 10 high-impact medical journals, including Circulation and the Journal of the American College of Cardiology.
They obtained information on 323 randomized, controlled, clinical trials in cardiology, gastroenterology, and rheumatology. Of these trials, just 45.5% were adequately registered, meaning they were registered before the end of the trial and the primary outcome was clearly specified. Among these trials registered adequately, investigators say that there were discrepancies between outcomes registered and outcomes reported in 31% of the published papers. Almost 83% of the differences in the primary outcomes favored reporting statistically significant results, according to Mathieu and colleagues.
Overall, more than one-quarter of the published studies were unregistered, 14% were registered after the study was finished, and 11% were registered with no description or an unclear description of the primary outcome.
“A main goal of trial registration is to enhance transparency of research and accountability in the planning, conduct, and reporting of clinical trials, an objective achieved by making available details about the trial,” write Mathieu and colleagues. “Therefore, adequate registration should be a safeguard against publication bias. A major step has been achieved with the ICJME initiative for trial registration, and the existence of all trials is now publicly available. However, after this first step, the quality and timing of registration still needs improvement.”
Speaking with heartwire, Ross said that he believes registries like ClinicalTrials.gov are excellent ideas and that the research profession is headed in the right direction by making the medical field more transparent. However, some changes are needed to make it more effective, including making more information mandatory, including specific details about primary and secondary end points, and better oversight.
“There aren’t a lot of resources being devoted to oversight right now,” he said. “Who’s going to be responsible for catching up with companies and investigators who are registering trials and not reporting or not filling in all the needed fields?”
************
Ross and Krumholz were previously consultants for the plaintiffs in litigation against Merck related to rofecoxib. Krumholz has research contracts with the American College of Cardiology and the Colorado Foundation for Medical Care, has served on the advisory boards of Amgen and UnitedHealthCare, and is the academic editor-in-chief of Circulation: Cardiovascular Quality and Outcomes, and Journal Watch Cardiology.