Psyche, Science, and Society

Les Roberts responds to criticism by Iraq Body Count

June 19th, 2006

Iraq Body Count (IBC) – as part of their effort to defend their work against critics ]including me: When Promoting Truth Obscures the Truth: More on Iraqi Body Count and Iraqi Deaths], who argue that their estimates of Iraqi civilian casualties are severe underestimates – launched a rather nasty attack on Les Roberts, lead author of the classic Lancet study of Iraqi casualties. They essentially accused Roberts of unethical behavior in an email which they forwarded to sources that had published work of his critical of IBC. Here is the IBC email. [Roberts response is below]:

From: John Sloboda [mailto:john@oxfordresearchgroup.org.uk]
Sent: Sunday, June 18, 2006 5:10 PM
To: les@a-znet.com
Cc: Charles.Hoge@NA.AMEDD.ARMY.MIL; hpn@odi.org.uk; cis_info@mit.edu;
tai@alternet.org; Francesco.Checchi@lshtm.ac.uk; Hamit Dardagan; Josh
Dougherty
Subject: your mis-citation of New England Journal of Medicine

Dear Dr Roberts,

This email concerns your mis-citation of a New England Journal of
Medicine article as well as a related error.

In your paper entitled “The Iraq War: Do Civilians Matter?”, published
in July 2005 by the MIT Center for International Studies in its series
“Audits of Conventional Wisdom”, you state that:

“A report in the New England Journal of Medicine in July 2004, based on
interviews with returning U.S. soldiers, suggests an unintentional
non-combatant death toll of 133 deaths per day.”
(http://web.mit.edu/cis/pdf/Audit_6_05_Roberts.pdf page 2)

This statement was repeated in the version of this paper republished on
the Alternet web site (http://www.alternet.org/story/31508/) and the
estimate of 133 violent deaths per day also appeared in Table 6 of a
publication entitled “Intepreting and using mortality data in
humanitarian emergencies” authored by Francesco Checchi and yourself,
and published by the Overseas Development Institute (ODI) as
“Humanitarian Practice Network Paper no 52″ (2005)
(http://odihpn.org/documents/networkpaper052.pdf page 30). The table is
entitled “Estimates of violent deaths per day in occupied Iraq” and the
source of the “133 deaths per day” estimate is described in that table
as “Mental Health Study, 2004″.

In all cases, the sole reference you provide for this figure of 133 is:

Hoge CW, Castro CA, Messer SC et al. “Combat duty in Iraq and
Afghanistam, mental health problems and barriers to care. “New England
Journal of Medicine. 351 (July 1. 2004): 13-21.” (NEJM)

As we pointed out in our paper “Speculation is no substitute: a defence
of Iraq Body Count” published in Arpil 2006
(http://www.iraqbodycount.org/editorial/defended), and which was copied
to you at the time, nowhere in this cited NEJM paper is there any
reference to an estimated per-day rate of violent deaths, whether 133 or

any other number, and there is nothing in any of your publications to
explain how this 133 per-day rate is derived (our discussion of this
matter is contained in section 3.4.2 (page 14 of pdf version) and in
greater depth in the related Appendix 3.4.2.a (page 38 of pdf version).
You simply cite the undoubtedly prestigious Journal as if it contained
the facts you claim for it, which it does not, and nor do you explain
how you take the data in that article and transform it into the “133
violent deaths per day”, which is surely the crucial calculation and
which would allow critical appraisal of your “133″ number.

We recently contacted the first author of the NEJM paper, Dr Charles
Hoge, who replied as follows:

“In no way can our data be used to estimate civilian deaths. We ask two
questions related to killing of enemy combatants and civilians on the
our anonymous questionnaire that we ask U.S soldiers, but neither can be

used to estimate casualty rates. We ask if at any time in the deployment

the soldier perceived that he was responsible for the death of an enemy
combatant and another similar question pertaining to the death of
civilians. Since all members of a team may in some way be responsible
during a combat operation these questions can in no way be used to
estimate actual civilian casualty numbers. ” (email to John Sloboda,
dated 8th May 2006)

In summary, you have published a claim, on the basis of the Hoge et al
paper, which the lead author of that paper says is unsustainable (just
as we had independently argued).

There are two matters of serious concern:

1. You have misused the authority of the New England Journal of Medicine

and the authors of this July 2004 paper to promote a claim which has no
basis in that study and which is explicitly rejected by the authors of
that study.

2. The supposed 133 per-day-rate of civilian deaths is one of several
“estimates” used by you and many of your readers to make unwarranted
claims about the relative value of different studies of Iraqi mortality,

and the likely overall death toll. Your use of this figure, and the use
made of it by others, has thus helped to spread confusion and
misinformation on a matter which is of the utmost gravity, and where
therefore the highest standards of rigour and professionalism are needed

from those claiming academic expertise and authority.

We note that you have made no attempt to correct these and a series of
other errors since receiving our “Speculation is no substitute” document

referenced above, wherein we subjected some of your claims about Iraqi
deaths and the sources of information on them to rational and critical
analysis. Since the critiques of “amateurs” apparently leave you
unimpressed, whether correct or not, we hope that you will show greater
regard for the opinion of Dr Hoge, whose paper you have misused.

We also note that, whereas after the intercession of co-author Francesco

Checchi you agreed to correct a blatant factor of two arithmetical error

about the daily rate of deaths recorded by Iraq Body Count, you have
made no discernible attempt to correct the exact same error in online
articles authored solely by yourself. Thus even the simple errors
accepted by yourself and corrected in ODI 2005 remain unchanged in MIT
2005 as well as in the (probably more widely read) version of the MIT
paper reproduced at Alternet.org. We expect that professional standards
would require you and other responsible parties to ensure that these
errors are corrected forthwith, without further delay.

In the light of the seriousness of this matter, we are copying this to
Dr Hoge, as lead author of the mis-cited NEJM study, your co-author
Francesco Checchi, the Humanitarian Practice Network of the ODI
hpn@odi.org.uk , the MIT Centre for International

Studies cis_info@mit.edu , and to the Managing
Editor of Alternet tai@alternet.org

Yours sincerely,

John Sloboda (on behalf of the Iraq Body Count Team)

This argument is silly. Of course Hoge’s data can be used to estimate civilian fatalities from American efforts. The accuracy of those estimates can be questioned, but to act as if they have no idea what Roberts is up too is more than silly. Assuming, as I do, that IBC are not stupid, it sure starts to look as if they are intentionally duplicitous.

This attack on Roberts is especially galling from an organization that admits their estimates are too small by a factor of about two and makes virtually no attempt to communicate this to those in the press who cite IBC’s lowball estimates as gospel.

Well, IBC seems to have picked on the wrong guy this time! Roberts relaxes his restraint and demolishes IBC’s claims and the accuracy of their estimates.

Dear Dr. Sloboda,

I was sorry to receive the e-mail below. In dozens of interviews where
reporters have delved into the discrepancies of our death tolls over the
first 18 months of the war in Iraq, I have attempted (sometimes
unsuccessfully) to emphasize my gratitude to IBC. IBC kept the issue of
civilian deaths in the public eye for months when insecurity kept me
from going to Iraq. I suspected we had the same motives and goals and
had hoped that our efforts might complement one another. I believe that
I emphasized this in my note to Hamit Dardagan on 10/28/04 and my
unanswered e-mail seeking cooperation with you 3 months later. But,
given the widely broadcast and personal nature of your comments below, I
am compelled to respond.

1) In the HPN report and the MIT Audits of Conventional Wisdom, I do
cite Dr. Hoge’s fine article on the mental health of returning soldiers.
In that article it is reported that interviewed Army soldiers returning
had been there 8 months and that (in Table 2) 14% of them reported
“Being responsible for the death of a non-combatant”. The Marine column
of this table is more complex, while it says 28% felt they were
responsible for killing a non-combatant, in the methods section their
duration of service is less clear. There were two Divisions sampled and
one was there for 6 months and one was there for 3 months. Thus, we
need to make some assumptions to acquire a daily death rate. These
include:
a) the interviewed Marines were in Iraq 4.5 months on average;
b) there were US 135,000 soldiers serving in Iraq during this period;
c) 70% were Army, and 30% were Marines. We felt that this was
conservative because other forces (Air Force) would probably have
reported higher rates and because it is believed the Marine fraction
averaged a little higher;
d) there were 30.4 days in a month.

The 133 number came from Richard Garfield almost 18 months ago. I have
not discussed this note or his assumptions with him since then and I
just today calculated an almost identical 137 deaths per day based on
the assumptions above. Of course Dr. Hoge did not say that this is
reported in his article. He is a highly regarded mental health expert
and I suspect that tallying death tolls is outside of the scope of his
ethical review for the research and perhaps his comfort level as well.
Moreover, if he said, “Yes, we reported a civilian death toll six times
higher than our Commander in Chief,” he would probably lose his job.

The text of the Hoge article implies that an individual being
interviewed felt responsible for a death. We shared Dr. Hoge’s concern
that it is in the best nature of humans to feel individual guilt for
something that was done by a group. Given that the bulk of civilian
deaths were from artillery and air-strikes according to our study and
the NCCI dataset, and that those soldiers would be unlikely to fully
witness their damage, this estimate was probably a gross under-estimate.
Also note that the Hoge article would record a soldier who thought they
killed 5 civilians the same way it would record a single killing. Thus,
attributing that 14% of Army soldiers accidentally killed “a” civilian
would err toward a low estimate.

2) Please note the Audit article says that the Hoge reference
“…suggests an unintentional non-combatant death toll of 133 deaths per
day.” In the HPN paper the table (6) citing the death toll estimates is
headed in bold as, “Violent deaths per day implied”. We have never said
“reported” or “recorded.” We used words such as “suggests” and
“implied” to convey the notion that we had some role in interpreting
these data.

Both of the HPN and Audit articles were meant to give an overview of the
issue of civilian death tolls. All of the time, public health data is
collected for one purpose and is later used for another. Most newspaper
reports cited in your data set were not collected to show a tally of
deaths nationwide. Given that a large sample of returning soldiers
reported accidentally killing civilians, and that this rate was
quantified from a sample, it would have been irresponsible to have not
mentioned the Hoge article in these overviews. Your assertion that we
“… have misused the authority of the New England Journal of Medicine
and the authors of this July 2004 paper…” is perplexing.

3) Please note that all of the mortality sources cited in the HPN paper
did not provide a daily violent death rate. Death rates were not
provided by IBC. In each case, assumptions needed to be made to get the
provided estimate. The assumptions regarding the Lancet estimate are
far, far greater than the Hoge article estimate. The Lancet estimate,
for example, assumes that no violent deaths have occurred in Anbar
Province; that it is fair to subtract out the pre-invasion violence
rate; and that the 5 deaths in our data induced by a US military
vehicles are not “violent deaths.” It seems strange that of the 5 other
references cited, you did not reach out to the authors of the highest
estimate, or the articles which required the biggest assumptions, but
instead reached out to a US military researcher who is the only one not
free to publish or speculate about the number of civilian deaths in
Iraq.

4) Most epidemiologists are reluctant to speculate about completeness of
their surveillance system before going out and trying to evaluate the
system sensitivity. It is easily done! You have publicly speculated
about IBC’s completeness rates, rates which are unheard of in times of
war. According to my colleague Riyadh Lafta (cc:ed above), only about
1/3rd of deaths were captured by the Government’s surveillance system in
the years before the Coalition invasion (that was based on a
conversation with Riyadh and the assumption that there were 24,000,000
people in Iraq dying at a rate of 5/1000/year, or that 120,000 people
were dying per year. In 2002, the Gov. of Iraq documented less than
40,000 from all sources).

First, given that there should be 120,000 deaths per year as a baseline
and every estimate I have seen implies that there are more violent
deaths than peaceful deaths, the IBC estimate seems very low.

Second, Jim Krane from AP News wrote me on March 2, 2006 regarding the
appearance that the IBC count might not be complete. (I have cc:ed him
above.) He points out that 24,000 violent death corpses have entered the
Baghdad morgue since the invasion when it used to be 2-3000 per year
before the invasion (thus~17,000 excess deaths as of 3 months ago). I
suspect that his suspicion was raised because less than a quarter of the
population lives in Baghdad, the violent death rate is lower there than
the rest of the country according to our data and NCCI’s data, and the
morgues traditionally capture a small fraction of deaths. Thus, twice
as many deaths in all of Iraq as recorded in the Baghdad morgue suggest
IBC is very incomplete.

Finally, we measured the sensitivity of your surveillance system during
the first 18 months, we found it was <5%. This is what we generously
referred to in the HPN paper as “cannot be more than 20% complete.” The
Falluja deaths in our data set were recorded by month and in IBC Falluja
deaths were not as distinct as elsewhere so that we could not match
them. However, among the other 21 violent deaths we encountered in our
random sample of 988 households, one was in the IBC data set. Thus,
unless you have evaluated the sensitivity of your system from some
independent data source, I hope you will temper the statements you make
about the complete nature of the IBC dataset.

5% completeness is the norm of newspaper reporting in times of war. (See
Patrick Ball’s work in Guatemala online with the AAAS) I suspect and
hope that the sensitivity has increased over time as systems have
improved and the role of major battles with airpower has diminished.
But, the speculation in the press that the real number might only be
twice the IBC tally is preposterous.

Last October 11th, I was invited at 2 hours notice before a flight left
to appear on the BBC program Newsnight with Jack Straw. I called you at
that time, hoping to hear that IBC had calibrated the system and to give
you the chance to defend the IBC sensitivity before saying on-air that I
had found it to be ~5% complete. Because we did not speak, I did not
then, and up until now, report our evaluation of your sensitivity in
public. I thought I was doing you a favor by calling.

5) As for your “Speculation is no substitute” paper, I discussed it with
some of my coauthors when it arrived. We decided that it was so devoid
of credibility, and so laden with self-interest rather than the interest
of the Iraqis, it did not merit a response.

6) It is unfortunate that the HPN paper had the wrong dates for the
17/death per day estimate of IBC. I believe that rate was from 2003.
That was a mistake on my part. If we revise the dates and rate to cover
the first 18 months of the war, the change from 17 to 27 deaths per day
does not change any conclusions ever presented in any paper or talk.
Surveillance is almost always incomplete in times of war compared to
surveys, and the conclusion remains the same, the three lowest estimates
of the death rate in Iraq are surveillance-based.

There will come a day when the foreign armies will diminish in Iraq and
a full census and accounting of this war will take place. I look
forward to that day.

I have been asked dozens of times, mostly by epidemiologists, what is
IBC’s motive to overstate the completeness of their estimate. I have
always been able to respond that I do not know. However, the focus of
your note below will only serve to bolster the universally unflattering
speculation.

Sincerely,

Les Roberts

This attack has convinced me that IBC is acting in bad faith. I had originally believed they were misinformed, being not particularly knowledgeable about epidemiology. But gradually this belief of mine has been challenged by IBC’s refusal to do anything to correct misuses of their data. Their response to critics, Speculation is no substitute: a defence of Iraq Body Count further disturbed me. It managed to make some valid points and correctly pointed out an error in a Roberts table while ignoring the main issues. This number was multiplied by that number. But they completely ignored the numerous sources of evidence that many, if not most, civilian deaths are not recorded in the press and, thus, in their database and estimates.

But this attack on Roberts is beneath contempt. Having met him once and having had a couple of brief email exchanges with him, I am extremely impressed with his concern for Iraqis combined with his sober scientific caution. When asked, he refused to speculate on issues, such as the number of Iraqi injuries, on which he did not have good data. In fact, I am so impressed with the Lancet study that I use it in my Social Research Methods courses to illustrate survey methodology, measurement validity, and a number of other concepts.

When the IBC response to critics appeared, I was disturbed. As a scientist, I was concerned that I might have made a major error. As I pursued their work, I gradually realized that it was smoke and mirrors, using weaknesses in critics’ work to deflect attention from the fact that, as they admitted, their estimates were way off [by a factor of two, as they admitted in an aside; by much more, in Roberts’ view].

After seeing this sleazy attack on Roberts, I am proud to have joined the chorus of IBC critics. At this point they give no sign of wanting to know what is happening in Iraq. Rather, they appear interested only in defending themselves at all cost to the truth. Too bad.

Entry Filed under: Iraq, Public Health, Research Methods, Social Issues, War and Peace