Psyche, Science, and Society

11 Comments Add your own

• 1. Robert Shone  |  February 28th, 2007 at 6:06 am

  Stephen Soldz wrote:
  >> A few of the details he presented reduced some of my
  >> qualms about the study

  Could you be more specific? What were your qualms, and how were they reduced?

  Stephen Soldz wrote:
  >> though I doubt that it will do much to calm the criticism

  Perhaps it would “calm” the criticism if it went some way towards *answering* it (rather than simply dismissing it). For example, will Burnham be releasing details of the sampling methodology used “in areas where there were residential streets that did not cross the main avenues in the area selected”?

• 2. joshd  |  February 28th, 2007 at 10:56 am

  “A few of the details he presented reduced some of my qualms about the study”

  I’d echo Robert in asking which details, and which qualms?

  “One thing that Burnham revealed is that they hope soon to release the identifier-removed data to a select group of qualified researchers.”

  This sounds kind of like there will be no ‘release’ of the data, but instead someday there will be a somewhat larger group of people in on the secret, after they select who they want to let in on it. Did you interpet this differently?

  Note too that Burnham contradicts himself again. He says the teams destroyed the lists of streets they made to pick from and says that he would have done this anyway because they promised to destroy any indentifying details, yet obviously the secret data they do have still has identifying details (which is the only way his statements about removing identifiers before they “release” the data could make sense).

  I think that was you at the end asking about ILCS, right? If so I’m wondering if you were satisfied with Burnham’s response to your points about ILCS and the child mortality issue. It seemed to me that your follow up points effectively rebutted the argument he had just given, and then he just evades it, while holding fast to the same conclusion that ILCS is wrong.

  Note that L2 requires it to be wrong by about a factor of 4. Note also that somehow L1, which is supposedly in perfect agreement with L2 (Burnham claims this repeatedly as strong confirmation for how right both surveys supposedly are), is not even a factor of 2 higher than ILCS (recall the debate over whether L1 and ILCS agree).

  How can L1 be so close to ILCS (where there could even be such a debate about them), and L2 so far from it (where even the notion of any such debate would be absurd), and it still be true that L1 and L2 have this seemingly perfect confirmation of each other that Burnham and Roberts claim for it? The answer is it can’t and doesn’t. The claim is deceptive and misleading. L1 and L2 measured very different events in very different proportions, and any credible analysis would suggest that a number of things have to be wrong with one or both of them. Since that truth is inconvenient in terms of their PR, they selectively cherry-pick one particular way to show you the data which conceals most of the relevant (and inconvenient) data and creates an illusion that they’re the same. Then this cherry-picked illusion is put onto impressive looking graphs and presented as “strong validation for both surveys”. Burnham misleads the audience with this claim several times, just as Roberts has in the past, and as their paper did.

  I was also a bit disappointed to see him choosing to poison the well with self-serving smears from Les Roberts when asked why he thinks IBC has been critical: “Les believes that Iraq Body Count which is dependent on donations and so forth has been damaged by other methods of reporting and that’s the basis of things”. Though I guess one could say that at least he had the good decency to distance himself from the baseless smear while poisoning the well with it.

  Also on IBC Burnham says he knows media surveillance is ‘not comprehensive’ (iow - it is “not unexpected” for IBC to be less than 10% of war deaths, as required by the L2 survey) because of data from “Central America”.

  Well, again Burnham is misleading the audience. We know what ‘data’ this vague but misleadingly authoritative-sounding reference is referring to. This is referring to the same one thing they’ve always cited: the Guatemala study by Patrick Ball that says 13 Guatemalan newspapers reported a very low percentage of rural conflict deaths in a few years in the early 1980s.

  But of course that anecdote, even if entirely accurate, is entirely inconclusive and inadequate to draw conclusions wrt what IBC will or won’t capture in Iraq, for a whole host of reasons. But what happens here is that Burnham is presented as this great expert, he says Central American data shows that IBC being 10% is normal, so people assume this is some kind of well-researched expert analysis of some large body of data when it’s simply one cherry-picked anecdote that gives them something they like, and which they therefore groundlessly present as applying to or having some kind of predictive power for, all conflicts past, present or future, and to any methodology involving media in some way or another (even if in a very different or much more comprehensive way, and in an entirely different kind of war or circumstance). Again, they mislead the audience for their own advantage.

  The reason why IBC disagrees with these guys, and has the “vociferousness” to say so, is not because of “funding” or any other self-serving smears Les Roberts chooses to “believe” and circulate, but has to do, at least in part, with the fact that they so persistently disinform people about all these issues in order to evade criticisms and make their own data and conclusions appear far more reliable than they actually are, which in turn requires the spurious and unwarranted dismissal, downgrading, or distortion of all other data (since no other data agrees with them, not even their own previous data, despite the continuous misleading claims to the contrary).

• 3. richard  |  February 28th, 2007 at 2:15 pm

  “…it’s simply one cherry-picked anecdote that gives them something they like…”

  joshd - do you have your own cherry-picked anecdotes then? I think most people would expect that media would under-report deaths in a civil war situation. They simply would not have the interest or werewithal to do anything else.

• 4. Robert  |  February 28th, 2007 at 2:37 pm

  joshd wondered:

  Note too that Burnham contradicts himself again. He says the teams destroyed the lists of streets they made to pick from and says that he would have done this anyway because they promised to destroy any indentifying details, yet obviously the secret data they do have still has identifying details (which is the only way his statements about removing identifiers before they “release” the data could make sense).

  Well, another way that statement could make sense is if the data included individual level identifiers such as sex, DOB, and date of death. In general, it’s quite common to combine ages into broader age groups if you don’t actually need the exact dates. Depending on the exact purpose of each study, other individual or household characteristics can also be combined into broader categories.

• 5. joshd  |  February 28th, 2007 at 4:41 pm

  “do you have your own cherry-picked anecdotes then?”

  I’ve been collecting quite a few cases, but there’s many different forms of media-based data on war deaths of varying comprehensiveness and in different types of wars and countries. There’s only a couple that are really comparable with IBC (the Guatemala thing is not one of them) and neither they, nor most of the ones that aren’t very comparable with IBC, show the same thing as the portion of the Guatemala data Les Roberts (and now Gilbert Burnham) cherry-picks and presents as a universal.

  “people would expect that media would under-report deaths”

  is a fairly reasonable assumption, and you could even quote IBC for the same assumption. However, the claim that it’s “not unexpected” for IBC to be 5 or 10%, or “Media reports always miss most deaths in times of war”, or “5% completeness is the norm of newspaper reporting in times of war. (See Patrick Ball’s work in Guatemala online with the AAAS)” are each very different claims than that, and are all groundless disinformation.

• 6. richard  |  February 28th, 2007 at 6:25 pm

  “…….groundless disinformation.”

  So you agree that media reports will be unreliable and under-report deaths by an unknown degree. Which means that the IBC numbers don’t have much value.

• 7. joshd  |  February 28th, 2007 at 11:13 pm

  No Richard, I don’t agree. But you already knew that. You did however create the appearance of a retort. Congratulations.

• 8. Robert Shone  |  March 1st, 2007 at 7:30 am

  Richard wrote:
  >> media reports will be unreliable and under-report
  >> deaths by an unknown degree. Which means that the
  >> IBC numbers don’t have much value.

  It depends how you measure “value”. If it requires a guarantee of “reliability”, then don’t expect any study (including Lancet) to “have much value”.

  Most of IBC’s critics who emerged after the MediaLens 2006 campaign (and who were silent on the issue before MediaLens made it an issue) seem to be agreed that IBC’s work had “value” in the “early days”. Then, suddenly, this “value” disappeared overnight when Lancet 2004 was published. IBC hadn’t changed anything - their figures met the same criteria, followed the same methodology, etc - but their “value” was suddenly lost.

  I suggest this says more about the narrow “value” judgements of IBC’s critics, than it does of the value of IBC’s work.

• 9. richard  |  March 1st, 2007 at 1:54 pm

  “I suggest this says more about the narrow “value” judgements of IBC’s critics, than it does of the value of IBC’s work. ”

  - Well, I’d say that is just par for the course. Data on a new subject (in this case the effects of the Iraq war on death rates) tend to be first viewed as having value, even if they later turn out to be less useful than at first thought. That is pretty common in science publications, for example. Should someone else carry out a still-larger study than Lancet 1 or 2 and come up irrefutably better estimates that happen to be much lower, you can be sure that the ‘value’ of Burnham’s work will drop quickly enough. Right now, though, it is hard to buy the argument that the IBC counts are superior to the Lancet survey data.

• 10. Donald Johnson  |  March 1st, 2007 at 5:03 pm

  I think some of us became disenchanted (rightly or wrongly) with IBC after they published their two year analysis in the summer of 2005. Previous to that I took their disclaimer at face value (media reports will only capture some of the deaths, such is the sad nature of war, etc…) What was startling about the IBC numbers were the extremely low figures for civilians killed by the coalition in most months (with the exception of March/April 2003 and a couple of months where Fallujah was invaded). I’m a little less certain about everything regarding Iraqi death statistics these days, but the idea that the US killed only 370 civilians in the third year (which was similar to the US-inflicted death rate in most months in the first two years) still seems very hard to swallow. Israel kills Palestinian civilians at a higher rate in many or most years since the intifada began.

  But speaking of that, to say something in defense of IBC, I’m a little tired of hearing about the low coverage of civilian deaths in other wars myself. The media figures aren’t always low, or else there are many other wars where the civilian death toll is much greater than anyone suspects. Nobody claims the I/P conflict has really killed 5 to 10 times the number actually reported dead. In some wars there’s a huge media undercount–in others maybe there isn’t, so far as we know.

  It’d be nice, though, if the media would make clear that when they compare IBC figures to the estimates made for other events (the number killed by Saddam, for instance), it’s apples and oranges as far as methodology is concerned.

• 11. David Kane  |  April 10th, 2007 at 8:36 am

  “there was no one among this highly qualified audience who thought the study was not highly credible.”

  For the record, I was in this audience and I do not think that the study is “highly credible” because of the 98% response rate. I do not know of a single nation-wide single-contact survey (in any country on any topic ever) with this high a response rate. The study might still be correct (and Burnham struck me as a highly credible researcher) but the study itself is not “highly credible,” in my opinion.

Leave a Comment

Some HTML allowed:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Trackback this post  |  Subscribe to the comments via RSS Feed