Psyche, Science, and Society

Iraq Body Count finds a task worth their time

October 24th, 2006

Because they are based on media reports, it is clear that Iraq Body Count’s reports of numbers of Iraqi civilians dead from violence are clearly an undercount. IBC points this out in occaisional asides. Nonetheless, the media have routinely reported (until very recently) IBC’s numbers as if they were the actual number killed in Iraq, rather than the number reported in the Western press. IBC usually claims they have no idea what the real number is and says “we don’t do estimates,” despite sometimes estimating that they believe they capture around 50% of the deaths.

Further, IBC has repeatdly claimed to be too busy to make any effort to correct these media distortions of their numbers. Nor could they be bothered to protest when the British Government incorectly cites their figures as “the” number of civilian dead in Iraq. They stand mute when President Bush cited a figure apparently coming from their estimates.

But, when higher estimates [of 600,000 dead] are published by some of the top epidemiologists in the area of mortality estimation in conflict situations, IBC sees red. There appears to be no limit to the time they will spend casting aspersions on the epidemiologists and their work. Not only do they publish an extensive Press Release that argues that the new estimates are likely wrong beacuse, well, because they are higher than IBC’s figures. [Yes, I', exagerating, but not by that much. They also argue that Iraq, torn as it is by civil war, must have government data systems accurately reporting deaths and injuries.] I hope to write more on the IBC critique in the future.

Now, I am not totally convinced by the new Lancet-published Iraq mortality study. But I think it is quality work requiring careful consideration, not hatchet jobs. And I think it, like all scientific studies, is not imune from examination and critique. But I do find it disturbing that IBC, which can’t be bothered to take any action regarding obviously false reports of its numbers, is obsessed to undermine an apparently credible study. Today brings further evidence that IBC has plenty of time to act on false reports, if only they saw the point. They are sending around to defenders of the Lancet mortality study another letter attacking the study and, implicitly, its defenders.

Here is the IBC letter:

Dear Professor [Name Withheld]

We note that you are a signatory to the article in “The Age” citing the recent Lancet estimate of 655,000 dead as “the best estimate of mortality to date in Iraq that we have, or indeed are ever likely to have.”

Are you aware of the much larger and more precise UNDP-funded survey which found a significantly lower number of war-related violent deaths in an overlapping period than is implicit in the present Lancet-published estimate? http://www.iq.undp.org/ILCS/overview.htm) If so, why have you disregarded its findings in favour of Lancet?

You go on to say “We urge open and constructive debate, rather than ill-informed criticism of the methods or results of sound science.” We welcome your call for open and constructive debate. As you may know we have published some quite widely reported reservations about the Lancet study (PDF attached – a balanced report on this is here: http://blog.washingtonpost.com/worldopinionroundup/2006/10/ is_iraqs_civilian_death_toll_h.html ). Further queries have been raised
in a recent Science Journal article (appended below).

We would be very grateful if you would let us know how, in particular, you would defend the study against these criticisms which we, and many others, believe cast serious doubt on the author’s claims that the study’s results can validly be extrapolated to provide a meaningful estimate for the whole of Iraq. We of course assume that you are fully conversant with the methods described both in the Lancet paper itself ( http://download.thelancet.com/pdfs/journals/0140-6736/PIIS0140673606694919.pdf ) and the lengthier descriptions given in supporting notes published by MIT ( http://web.mit.edu/CIS/pdf/Human_Cost_of_War.pdf ).

Best Regards,

John Sloboda, FBA.
Co-founder Iraq Body Count
Director, Oxford Research Group
Professor of Psychology, Keele University

Science 20 October 2006:
Vol. 314. no. 5798, pp. 396 – 397
DOI: 10.1126/science.314.5798.396
News of the Week

EPIDEMIOLOGY:

Iraqi Death Estimates Called Too High; Methods Faulted

John Bohannon

A new estimate of the number of Iraqis who have died as a consequence of the U.S.-led invasion in March 2003 has ignited a firestorm of its own. At 400,000 to 800,000 deaths, the new number is at least 10 times higher than estimates cited by the Iraqi government and U.S.- led coalition. U.S. President George W. Bush immediately dismissed the study, characterizing its methodology as “pretty well discredited.” Other Administration officials charged that the study, released with significant publicity 4 weeks before U.S. midterm elections, was politically motivated. Researchers who spoke with Science disagree that the authors’ motives are suspect but raise several questions about the methodology of the study, which was published 11 October in The Lancet.

Experts on both sides of the debate concede that it is notoriously difficult to get an accurate count of casualties in Iraq. The Iraqi
Ministry of Health has estimated up to 40,000 violent deaths so far, based on death certificates reported by hospitals and morgues. That figure falls within the range published by Iraqi Body Count, an independent London-based group opposed to the war that compiles casualty numbers from media reports. There is little doubt that the real number of deaths is higher than this, because only a fraction of deaths are officially recorded or reported by journalists. But just how small is that fraction?

The Lancet study, designed by researchers at Johns Hopkins University in Baltimore, Maryland, is based on a survey conducted between May and July by a team of 10 Iraqi health workers. (The Johns Hopkins researchers met with the Iraqi team twice across the border in Jordan to advise on the survey techniques.) The team visited 47 neighborhoods in 18 different regions across the country, going door- to-door and asking families about recent deaths. They collected data from a total of 1849 households
containing 12,801 residents. For the 14 months before the invasion, the Iraqi families reported 82 deaths, an annual death rate of 5.5 per 1000 people. Within the same households, 547 people died between the start of the invasion and July of this year–an annual increase of 7.8 deaths per 1000. By applying this rate to the entire population of 27 million, the researchers conclude that
655,000 more Iraqis have died than would have if the invasion had never happened. About 8% of these extra deaths are attributed to deteriorating public health, but an estimated 601,000 are violent–56% from gunshots and about 13% each from air strikes, car bombs, and other explosions. The researchers calculate a 95% probability that the true number of violent deaths lies between 426,369 and 793,663.

Many academics spoke up in defense of the study. “I too find the survey’s estimates shockingly high, … [but] the choice of method is anything but controversial,” wrote Francesco Checchi, an epidemiologist at the London School of Hygiene and Tropical Medicine on 12 October on a humanitarian Web site. The statistical technique used, called cluster surveying, divides the population into different regions, neighborhoods, and households, in contrast to a random sampling of people on the streets.

The method may be sound, but several critics question the way it was carried out in this study. Madelyn Hicks, a psychiatrist and public health researcher at King’s College London in the U.K., says she “simply cannot believe” the paper’s claim that 40 consecutive houses were surveyed in a single day. “There is simply not enough time in the day,” she says, “so I have to conclude that something else is going on for at least some of these interviews.” Households may have been “prepared by someone, made ready for rapid reporting,” she says, which “raises the issue of bias being introduced.”

Lead author Gilbert Burnham, an epidemiologist at Johns Hopkins, counters that “40 adjacent households is entirely achievable in a day’s work if well organized.” Les Roberts, also at Hopkins, adds that 80% of the 547 deaths were corroborated with death certificates. The fact that hundreds of thousands of death certificates seem to have gone unregistered by the Ministry of Health is no surprise, says Roberts, because “those have always been grossly underreported.”

Neil Johnson and Sean Gourley, physicists at Oxford University in the U.K. who have been analyzing Iraqi casualty data for a separate study, also question whether the sample is representative. The paper indicates that the survey team avoided small back alleys for safety reasons. But this could bias the data because deaths from car bombs, street-market explosions, and shootings from vehicles should be more likely on larger streets, says Johnson. Burnham counters that such streets were included and that the methods section of the published paper is oversimplified. He also told Science that he does not know exactly how the Iraqi team conducted its survey; the details about neighborhoods surveyed were destroyed “in case they fell into the wrong hands and could increase the
risks to residents.” These explanations have infuriated the study’s critics. Michael Spagat, an economist at Royal Holloway, University of London, who specializes in civil conflicts, says the scientific community should call for an in- depth investigation into the researchers’ procedures. “It is almost a crime to let it go unchallenged,” adds Johnson.

Co-author Roberts is no stranger to such controversy. He led a smaller study of Iraqi casualties, published in The Lancet in 2004, that estimated 100,000 deaths. That work was criticized for relying on too few samples. This time, he says, “we took enough samples, and if anyone wants to verify our results, it’s easy.” The study suggests that close to four times the number of deaths occurred in the first half of 2006 than in the first half of 2002, he says, “and anyone could simply pick four to six spots in Iraq and go to the local graveyards. The increase … should be obvious.”

For now, Spagat says he is sticking with casualty numbers published by the United Nations Development Programme (UNDP). A UNDP survey of 21,668 Iraqi households put the number of postinvasion violent deaths between 18,000 and 29,000 up to mid-2004. “When a survey suggests so much higher numbers than all other sources of information,” he says, “the purveyors of this outlier must make a good-faith effort to explain why all the other information is so badly wrong.”
–

John Sloboda
Executive Director

IBC now clearly sees discrediting rivals as their major activity. Correctly reporting the number of Iraqi deaths ceased mattering to them ages ago. I used to see IBC as misguided. I have long ago changed that opinion.

Entry Filed under: Iraq, Middle East, Mortality, Public Health, Research Methods, Science, Social Issues, War and Peace