Psyche, Science, and Society

Possible flaw in Lancet study worth considering

October 20th, 2006

Here is a note I posted on the Media Lens Message Board in response to heated controversy around the claims of several British scientists to have found a flaw in the sampling methodology in the new Lancet Iraq mortality study [I've posted the piece describing those scientists' ideas below my comments]:

Folks,

I’m not known as a defender of IBC. And I think their recent citique of the Lancet study is weak. But I think the issue raised here of “main street bias” is a serious one and should be taken seriously. I had not wanted to write anything till I could write a comprehensive article, but am disturbed by the tone here.

IBC is right in (at least) one respect. Studies that result in surprising findings should be subject to careful scrutiny. Of course, it’s especially difficult in the present situation where major policy implications flow from one’s critique. It’s also difficult in the gotcha environment where Josh D. jumps on every statement to essentially accuse Lancet defenders of bad faith and its authors of fraud while those who support the study jump on IBC.

While I personally find certain things that IBC has done to be despicable [not making a much greater effort to correct media misreporting of their numbers; likening critics to terrorists; the tone of the Lancet critique), that is irerlevant to the question of the quality of the Lancet study.

The Lancet numbers are surprising. I think the study is of high quality. Yet, that doesn't mean one shouldn't look for flaws. Most of the one's mentioned, such as too few clusters are silly. As many noted, the number of clusters affects the size of the confidence intervals. Also, the relatively low design effect (1.6) suggests this isn't a problem. Of course more clusters are better. But, when designing and conducting a survey, one always balances costs and benefits. The balance in the study is reasonable.

But the possibility of a subtle bias in the sampling design is the crucial issue. If the sample is unbiased, it is hard to come up with an explanation of how such a mortality rate could arise without fraud, which I doubt. [I thought carefully about this last night and think there's enough evidence in the study to rule out [CORERCTED 10-20-2006 1:23EDT] explicit fraud by interviewers at all likely.]

I have seen speculation about some bias introduced regarding these procedures. I’m not at all sure that these authors are correct. But I do think their point needs to be answered. If the sampling strongly over-included main streets, I could imagine it biasing the study.

Unfortunately, I will be out of town for the weekend and away from computer. But I did want to weigh in and say “Cool it folks.” I’m an antiwar activist, but I’m also a scientist. If the study is right, I want to know it. But, if its wrong, I also want to know that.

I also agree with IBC that the number of casualties in Iraq is too high, whether 100,000 [IBC numbers times 2 for nonreporting bias that they admit, andJon Pederson's: estimate] ] or 655,000. I believe the magnitude makes a difference. But I also want to find the truth, whatever it is. I hope others will join in that.

****** Lancet Critique *****
Lancet study fundamentally flawed: death toll too high
October 19, 2006 – 1 page – For immediate release:

Researchers at Oxford University and Royal Holloway, University of London have
found serious flaws in the survey of Iraqi deaths published last week in the Lancet.

Sean Gourley and Professor Neil Johnson of the physics department at Oxford University and Professor Michael Spagat of the economics department of Royal Holloway, University of London contend that the study’s methodology is fundamentally flawed and will result in an over-estimation of the death toll in Iraq.

• The study suffers from “main street bias” by only surveying houses that are
located on cross streets next to main roads or on the main road itself. However
many Iraqi households do not satisfy this strict criterion and had no chance of
being surveyed.
• Main street bias inflates casualty estimates since conflict events such as car
bombs, drive-by shootings artillery strikes on insurgent positions, and market
place explosions gravitate toward the same neighborhood types that the
researchers surveyed.
• This obvious selection bias would not matter if you were conducting a simple
survey on immunisation rates for which the methodology was designed.
• In short, the closer you are to a main road, the more likely you are to die in violent activity. So if researchers only count people living close to a main road then it comes as no surprise they will over count the dead.

During email discussions between the Oxford-Royal Holloway team and the Johns
Hopkins team conducted through a reporter for Science, for an article to be published October 20, it became clear that the authors of the study had not implemented a clear, well-defined and justifiable methodology. The Oxford-Royal Holloway team therefore believes that the scientific community should now re-analyze this study in depth.

The team can be reached for comment at;
Gourley: s.gourley1@physics.ox.ac.uk
Johnson: n.johnson@physics.ox.ac.uk
Spagat: M.Spagat@rhul.ac.uk

Entry Filed under: Iraq, Middle East, Mortality, Public Health, Research Methods, Science, Social Issues, War and Peace