Iraq Body Count trashes Lancet Iraq mortality study yet again
April 7th, 2007
The Iraq Body Count (IBC) folks have written a letter to the World Socialist Web Site critiquing the site for having referred in a recent article to the 600,000+ excess Iraqi deaths estimated in the October 2006 Lancet Iraq mortality study. The IBC letter is rather silly, objecting, for example, to the fact that the mortality study was a “cluster survey”:
Secondly, the Lancet researchers visited 47 neighbourhoods and conducted interviews in 40 adjoining households in each neighbourhood. Only about 1,800 households containing 12,000 Iraqis were surveyed. These households reported a total of 302 violent deaths, each of which has been multiplied by two thousand to provide an estimate of how many of Iraq’s estimated 26 million population would have died if this proportion of deaths were representative of the country as a whole.
The IBC folks contrast the dead counted in the mortality survey with the dead reported in the media that they count:
The Lancet researchers documented only 300 violent deaths. Iraq has reached such a sorry state that IBC records 300 deaths every few days.
Correct. So what’s the point here? Is it better to precisely count the wrong thing (deaths reported in the media) than to use survey methodology to somewhat imprecisely count what one is interested in knowing? The IBC folks sure seem to think so.
IBC also repeats another faulty argument they have previously made:
The study’s central estimate of over 600,000 violent deaths seems exceptionally high. Even its lower bound 95 percent confidence interval of 426,000 violent deaths is shockingly high. It is very unlikely that incidents of this scale would be so consistently discounted by the various media in Iraq. Although IBC technically requires only two sources for every corroborated death in its database, we actually collect, archive and scrutinize every single report we can find about each incident before it is added to our database. For larger incidents, the number of reports can run into the dozens, including news published in English, in the original, and others, mostly the Iraqi press, published in translation. In IBC’s news archive for August 2006, the average-size attack leaving 5 civilians killed has a median number of 6 reports on it.
Their argument on the number of press reports for large incidents would only be valid if the press covered the whole country. Yet, there is a disproportionate proportion of the press around Baghdad and, to a certain extent, a few other cities. So, their claim here can also be understood as being due to this press running to the same few incidents to which they have access.
Further, the IBC claim here ignores the Lancet mortality study finding that the largest cause of deaths, indeed 56% of them, were due to gunshot wounds, which, by their nature, are not necessarily grouped in large incidents with multiple press reports.
The IBC folks make another rather odd point:
We would hope that, before accepting such extreme figures, serious consideration is given to the possibility that the population estimates derived from the Lancet study may be flawed. The most likely source of such a flaw is some bias in the sampling methodology such that violent deaths were vastly over-represented in the sample. The precise potential nature of such bias is not clear at this point. But to dismiss the possibility of such bias out of hand is surely both hasty and irresponsible.
I agree that “serious consideration is given to the possibility that the population estimates derived from the Lancet study may be flawed.” I further agree that studies with “extreme figures” should be examined closely. I also agree that sampling bias is the most likely place to find potential bias. And of course, one should not “dismiss the possibility of such bias out of hand.” But I further believe that seeking to dismiss a major study by pointing to”some bias in the sampling methodology” when “the precise potential nature of such bias is not clear at this point” is indeed “surely both hasty and irresponsible.”
The main sampling bias that has been raised so far is the so-called main street bias (MSB), having to do with alleged oversampling of streets intersecting main streets. It is possible that there may be some degree of such sampling bias, though the mortality study authors strenuously deny this. . But the MSB authors derive a bias factor of three by making extreme, unsupported, assumptions regarding their mathematical model’s basic parameters. Even if one were to make these extreme assumptions, a bias of three would still leave excess Iraqi deaths at over 200,000, way above the 60,000 that these authors from IBC appear to be defending. Any reasonable estimate of these parameters would lead to a much smaller bias factor and an estimate of excess deaths considerably higher.
It is possible, of course, that there are other, so far undiscovered, sampling biases in the Lancet study. As every scientist knows, any study may have unknown biases. But the possibility of unknown biases is not a plausible nor honorable reason to dismiss an apparently well-conducted study.
While encouraging skepticism about the Lancet mortality study figures, these IBC authors neglect to mention the virtual certainty that the IBC numbers are too small. After all, it strains credulity that, in circumstances of war, that all deaths would be reported in the media. In fact, IBC founder John Sloboda said in an April 2006 BBC interview:
We’ve always said our work is an undercount, you can’t possibly expect that a media-based analysis will get all the deaths. Our best estimate is that we’ve got about half the deaths that are out there.
I guess the fact that IBC counts are underestimates doesn’t fit the “message” IBC so carefully strives to create and so can safely be neglected when criticizing others. As far as can be told, IBC makes almost no effort to correct mainstream media’s routinely missing the fact of IBC’s numbers being undercounts of the true number of deaths. IBC routinely pleads lack of resources to correct such misinterpretations. It is to be wished that IBC would spent even a tiny fraction of the energy they’ve devoted to attacking the Lancet mortality study (see, e.g., here, here, and here), its authors and supporters, to correcting media misrepresentations of the meaning of IBC figures.
Skepticism, debate, and careful examination of potential biases is the heart of science. I maintain somewhat skeptical about the results of the Lancet mortality study. The results are so extreme that they deserve to be carefully examined. But I see no grounds to dismiss the study; the possibility of possible unknown bias is far from a reasonable basis for such dismissal. Amazingly, given the horrific conditions, other groups still are routinely conducting surveys in Iraq. It strains understanding why none of them has used the opportunity to provide an independent assessment of excess mortality. For replication is the only way that scientific debates are ultimately resolved. In the meantime, given all the evidence available at this point, it seems very likely that Iraqi excess deaths are in the hundreds of thousands, perhaps the high hundreds of thousands. Far too many have died as a result of the invasion and occupation and far too many will continue dying as the occupation and resultant resistance and civil war grind on.
NOTE: Tim Lambert at Deltoid has too additional items of interest. First, Les Roberts and Gilbert Burnham respond to a critical article in Nature. Second, there is an announcement that Lancet mortality study author Riyadh Lafta will speak this month at Simon Fraser on Friday, April 20, at 7 pm. Evidently, the United States denied Dr. Lafta a visa. I sure hope that his talk will be made available on the web.
Entry Filed under: Iraq, Middle East, Mortality, Public Health, Research Methods, Science, War and Peace
12 Comments
1. joshd | April 7th, 2007 at 9:55 pm
“But the MSB authors derive a bias factor of three by making extreme, unsupported, assumptions”
I’ll leave aside most of your post here (you start with a straw man in your first paragraph and continue with more I’ll get to below), but what are these “extreme” assumptions Steven? Tim Lambert floated this line in one of his pieces and was torn to shreds by one Robert Shone for failing to clarify what exactly are the “extreme” assumptions or how he was covertly distorting and changing all the “mathematical model’s basic parameters” in order to dismiss the MSB authors’ arguments.
You swallowed that line whole immediately on his blog and you just repeat it by assertion here. This is anything but “skepticism, debate, and careful examination of potential biases”. You’re avoiding exactly that with these entirely un-careful assertions.
You then pound away at this straw man theme:
“not a plausible nor honorable reason to dismiss”, “I see no grounds to dismiss the study”, “far from a reasonable basis for such dismissal”
Can you point to anything in this letter from Bulent and Lily that demanded that WSWS “dismiss” the study? If not, why are you writing this straw man rhetoric?
You have this exactly backwards. I think they saw WSWS promoting it as an established truth, “dismissing” the potential for bias like MSB or any number of other problems that can affect a study of this kind, particularly under such conditions, and their letter was appealing to them not to do this.
Why does WSWS “dismiss” Pedersen’s ILCS for example – which they have to do in order to present L2 as certain truth as they’ve been doing? One good guess is that they are so well informed on this issue that they have no idea it exists, as they write that the Lancet is “the only scientific investigation of Iraqi casualties”. Perhaps they should spend less time lecturing people, like Bulent and Lily, who actually study this issue and do some of it themselves.
But assuming they even know it exists, why no blog post from you decrying them for their irresponsible “dismissal” of an apparently well-conducted study? At least that posting wouldn’t have been a straw man.
2. Kevin Donoghue | April 8th, 2007 at 4:32 pm
Josh wrote: Tim Lambert floated this line in one of his pieces and was torn to shreds by one Robert Shone for failing to clarify what exactly are the “extreme” assumptions or how he was covertly distorting and changing all the “mathematical model’s basic parameters” in order to dismiss the MSB authors’ arguments.
What on earth is this supposed to mean? The parameters of the MSB model are entirely arbitrary. If the authors can pluck their parameters out of their asses why shouldn’t Tim Lambert emulate them? Maybe you thought Robert Shone won that argument. I thought he stalked off in a huff.
The whole MSB discussion is nonsense. There’s generally no point in constructing a mathematical model unless you have some data which you can use to provide estimates of the parameters. (Now and then a purely theoretical model is useful for clarifying issues but the MSB model clarifies nothing at all.)
Why does WSWS “dismiss” Pedersen’s ILCS for example – which they have to do in order to present L2 as certain truth as they’ve been doing? One good guess is that they are so well informed on this issue that they have no idea it exists, as they write that the Lancet is “the only scientific investigation of Iraqi casualties”.
How many times does this have to be explained? We don’t know how many deaths in total were recorded by the ILCS; we don’t know how many are in categories other than infants, maternity-related and “war related” deaths; and we don’t know what that particular term means. We don’t know very much about when the deaths took place or about their geographical distribution – if we knew more about those things we might have some insight into the Iraqis’ interpretation of “war related”. We might then be able to say whether the ILCS conflicts with Burnham et al. As things stand we don’t actually know that it does.
As for being scientific: the ILCS reports that N = 23,743 and in effect N is defined as whatever it is that the ILCS reported. Deaths are “war related” if the respondents said so. It’s a bit like saying IQ measures intelligence and defining intelligence as being whatever it is that IQ measures. I wouldn’t call that good scientific procedure myself.
Anyway shouldn’t the results of a scientific study be published? We hear a lot of whinging from the Lancet-bashers about how Burnham hasn’t published his data – isn’t it about time Pedersen at least published his friggin’ results? What’s the good of saying that the ILCS is “apparently well-conducted study” when we know only a little of what it found?
3. joshd | April 8th, 2007 at 6:35 pm
Kevin, your “explanation” about ILCS is rubbish. You can explain it until you are blue in the face and it won’t make it any less rubbish and wishful thinking on your part.
Violent deaths like those IBC or Lancet records are very obviously “war related”, except perhaps some of the criminal murders. People being shot and blown up in political violence and attacks are obviously war related and anyone seeing the questions in ILCS would know this. Iraqis are not idiots and you invent and exaggerate some great confusion around that term in order to _dismiss_ (take note Steven) ILCS.
You say that “We don’t know very much about when the deaths took place or about their geographical distribution”
The first is irrelevant as there was no war prior to the invasion and so obviously the “war related” deaths occurred between the invasion and the time of the survey. And ILCS gives regional info. L2 gives province info. The latter is slightly more detailed on geography, but not much.
You simply speculate some huge bias resulting from some supposed great confusion over the term “war related”, and you use this “extreme, unsupported, assumption” to “dismiss an apparently well-conducted study”.
I’d like to see more stuff released for ILCS. That guy David Kane should try writing Pedersen. I bet he’d have more luck than with his attempts to get stuff out of LR.
But we know that ILCS has a larger and better distributed sample, better supervision, and more crucially, it has a truly random sample, and that L2 does not. You choose to think that irrelevant and invent some great confusion over a term whose meaning is apparent to all of us, and would be even more so to those responding.
4. Kevin Donoghue | April 9th, 2007 at 4:42 am
Stephen,
I gather from SFU that a DVD will be made of Dr Lafta’s lecture but I wouldn’t count on it being available on the web. As for the visa, it seems to have been a deniable denial – the State Department messed him around for months and then told him to apply again. But there is little doubt at SFU that the obstruction tactics were deliberate.
Josh,
Your response (8th April) is vacuous. The facts are as I have stated and no amount of windy verbiage about wishful thinking will change that. Of course you will go on saying that people aren’t responding to your oh-so-compelling ILCS argument, ignoring the fact that I have done so repeatedly and others have too.
5. David Kane | April 10th, 2007 at 8:29 am
“That guy David Kane” here. Hello to all my Lancet friends! And to Stephen, my lunch partner today.
I am eager to contact Jon Pedersen, but I don’t know how. Suggestions welcome.
By the way, if any of you are ever in Cambridge, MA, lunch is on me.
6. Donald Johnson | April 11th, 2007 at 4:50 pm
As I recall, Pedersen was quoted at this very blog saying that the ILCS teams didn’t go to Fallujah in the spring of 2004 because it was too dangerous and he thought they probably avoided the more violent Shiite areas in the same time for the same reason. So it’s definitely an undercount if they avoided violent areas.
7. Robert Shone | April 13th, 2007 at 3:18 am
There’s an interesting thread about Stephen Soldz’s blog entry at Media Hell: http://www.mediahell.org/messageboard.htm
Kevin Donoghue wrote:
> Maybe you thought Robert Shone won that argument.
> I thought he stalked off in a huff.
If “stalked off in a huff” means abruptly ceasing to post, then I think it applies more to yourself, Stephen and Tim Lambert on the Main Street Bias issue. The questions that remain unanswered by the three of you remain:
1. What exactly were the alleged “wild assumptions” made by the MSB authors? (Note that Tim Lambert’s response that they interpreted “main street” conservatively is not a wild assumption of the MSB guys – it’s an opinion of Tim’s, which I’ve addressed [see link below] and which Tim has not responded to. Or to use your colloquialism, he stalked off in a huff.)
2. Where is the “redefinition” of f(o) which Tim says he provided, and which is necessary to understand how Tim reached his absurd value of 2/16 – one of the main parts of his “critique” of MSB? Nowhere did Tim indicate how he calculated 2/16, or how anyone could produce a value based on his “redefinition” (since his “redefinition” doesn’t exist).
3. How do the Lancet team support their assertion that “all” households were included in the sampling? (Giving a value of n=0 in the MSB equation). What sampling procedure did they use to give this outcome, and why haven’t they published any details on it?
Links relevant to above points 1 & 2:
http://scienceblogs.com/deltoid/2007/03/london_times_hatchet_job_on_la.php#comment-368483
http://scienceblogs.com/deltoid/2007/03/london_times_hatchet_job_on_la.php#comment-368459
8. Kevin Donoghue | April 16th, 2007 at 8:20 am
Robert Shone: Note that Tim Lambert’s response that they interpreted “main street” conservatively is not a wild assumption of the MSB guys – it’s an opinion of Tim’s, which I’ve addressed … and which Tim has not responded to.
Here’s how you addressed it: If Riyadh Lafta wanted to make sure “all” households were selected, then the selection process described in the Lancet would have been utterly useless.
Really? Can you provide a proof of that claim, please? Send a copy to the MSB people, they need it badly. At the moment all they have is a proof by repeated assertion. I’ve looked at street layouts in areas that I know well and the Lancet procedure as I understand it gives reasonable results in most cases. (It’s probably inferior to GPS, which also has problems; nothing is perfect.) There are layouts in which there is a bias towards main streets (q>1) but there are also layouts in which there is a bias towards back streets (q
9. Kevin Donoghue | April 16th, 2007 at 11:09 am
My comment got cropped; continues: …bias towards back streets (q
10. Kevin Donoghue | April 16th, 2007 at 11:12 am
Try again; continues: …bias towards back streets (q less than one). Bear in mind that the sampling does not begin on a main street or a back street, so that can go either way. What layouts are most typical in Iraq I don’t know. But Riyadh Lafta presumably does and since he was the field manager it’s uncharitable to assume that he didn’t give the matter some thought. The MSB team assume that he was so negligent that he excluded 91% of the population. I think that’s a wild assumption; YMMV. (Of course he couldn’t hope to exclude 0% either, if that’s all you want to say.)
I’ll leave it at that since the technology seems to have turned hostile.
11. Robert Shone | April 17th, 2007 at 10:27 am
Kevin, by definition, the sampling procedure as published by the Lancet authors wouldn’t include “all” households. The Lancet authors said that by “main streets” they meant “major commercial streets or avenues”. What proportion of all streets (including back alleys) would fall into that category? What proportion of “all” households would be sampled from roads which intersected those “major commercial streets” (even allowing for a spillover)? Obviously not 100%. Yet the Lancet authors claim “all”, ie 100%. The burden is clearly upon them to demonstrate how they can assert “all”.
12. Kevin Donoghue | April 17th, 2007 at 2:43 pm
The Lancet authors said that by “main streets” they meant “major commercial streets or avenues”.
So you keep saying. As does Anna L. Plurabella. Are you relying on each others’ testimony? Or on the testimony of the MSB squad? They also make this claim, I notice, but on a web page which might politely be described as propaganda. I’ve not heard or seen any comment by the Lancet authors to that effect.
If Burnham et al. ever did say that, it was indeed an extraordinary thing to say. It would be hard to find a major commercial street in Fallujah these days. I’ll ponder the implications if you can point me to the text in which the statement appears.