Stephen, I think you’re one of the more moderate and intelligent commentators on these issues, and I appreciate your taking the time to discuss them with me. But I must take issue with the above.

I’ve received four emails from Stephen during a correspondence I initiated (in which I sent a total four emails in reply to Stephen’s four – plus one postscript containing a link to an article I thought he might find useful).

As for my “too busy” comment, it was in response to an email from Stephen which raised many issues, including “extreme assumptions” in the MSB work, instability over time of variables which affect whether a given area is a “safe zone” or a “danger zone”, the above reference to whether I quoted him in full to Pedersen, his (Soldz’s) agreement on the importance of revealing errors in L2, his reasoning on why he thinks IBC are “wrong” on various points, several other points concerning IBC, and other points concerning questions over L2 etc.

It was to all the above that I had no time to respond in full, as I was “too busy” (but I did say I would respond in a few days time – I sent that on 5/12/06). And now I’ve responded in full (at least to the Pedersen email thing), as promised, a few days later.

Nothing else that you wrote in your above article mitigates this overstatement (and I think “overstatement” is putting it charitably). You should know that such statements are quickly quoted across the web (Tim Lambert wasted no time quoting you on Deltoid, for instance).

Your last sentence is hypothetical (and therefore irrelevant to your earlier assertion): “In any case, he thought such a “bias”, if it had existed, would affect results only 10% or so.” [my bold]

Stephen Soldz wrote:“Robert, did you ask his opinion on my entire paragraph on the issue, or only on the phrase you extracted, which is taken out of context.”

Stephen, I didn’t “extract” anything “out of context”. My email to Jon Pedersen (as well as providing a link to your piece), included both your emphatic statement:

“Pedersen did NOT think that there was anything to the ‘Main Street Bias’ issue.”

And also your statement immediately following it:

“[Pedersen] agreed, I thought, that, if there was a bias, it might be away from main streets [by picking streets which intersect with main streets].”

My query was specifically about the contradiction between the above two statements (“Given that the MSB criticism in fact addresses precisely the bias resulting from selection of streets *intersecting* main streets, the above two attributions of your views from Soldz seem contradictory.”)

If you were unsure either of what Pedersen was saying, or of what main street bias says, then perhaps you shouldn’t have been so emphatic in saying that Pedersen “did NOT” think there was “anything” to the main street bias “issue”.

I cannot interpret your statement

…based on what Pedersen has supposedly said to Soldz (Pedersen has already said Soldz didn’t get his views on MSB quite right).

as making a claim slightly stronger than one of “nuance.”

I agree totally that nuances can be distorted. That was why, in my original account, I placed lots of hedge words, such as “I think,” to indicate that it was my version. However, I also took the precaution of sending the comments to Jon to give him the opportunity to correct me, if he so chose.

I do wonder if Pedersen was reminded by Shone of my entire statement on MSB when he was solicited to comment on it. Again, a matter of nuance. [I put this to Robert Shone, but, suddenly, after sending me several emails a day, he became "too busy" to respond to my simple inquiry.]

Thanks for posting this Stephen. I’d point out however, that i did not say or suggest that you “grossly misrepresented” anything. What I was saying was that nuances can be lost or get twisted around when one person is describing a complicated discussion with someone else from memory. As such, I don’t take the exact phrasings you chose to use as if these were direct quotes of Pedersen, as some others have done. That is all.

“In Lancet 2004, something like half the non-Fallujah violent deaths were from April on.”

is incorrect, or at least an exaggeration. (This is the problem of writing from memory.) First, I do agree that the non-Fallujah comparison is the correct one, and is the one I intended. If my count on their Figure 2 is accurate (my eyes are old enough that I can’t quite see the red), 13 deaths occurred in the 13 months from March 2003 to March 2004 while 8 deaths occurred in the 6 months from April 2004 through September 2004. 8 is under half, though it is at a somewhat elevated rate, which is what I must have remembered.

As to where I got my statement

“Even more relevant is that the ILCS fieldwork in both Fallujah and the Shia south was completed before the fighting. that is, by early April 2004″

My source is, of course, Jon Pedersen. Here is an excerpt from an email from him on May 2 2006. My question in NOT bolded, his response is:

This timing of the fieldwork also matters as many people died in these fights, including many hundreds [600+] in Falluja and an unkown number, possibly over 1,000 in the South. Thus, results could be affected also by the time that fieldwork was conducted, even if there was perfect cooperation. Thus ILCS estimtes only 3,686 deaths in the whole Iraq Center, including al-Anbar. Whether the 600+ deaths in Falluja, the deaths in South, and other deaths were counted would affect estimtes to some degree, perhaps 10% or more. Is there any further information available as to when the bulk of the fieldwork was conducted in various governorates?

Acutally, the interviews within the closed zone were carried out before the closure, so it was not really much of a problem. Obviously, Falujah deaths are not counted into the estimate. (on general principles I would probably have reloied on other soiurces for those deaths anyway, because the survey was not designed to take represent such small areas accurately. As in the lancet study, Falujah contributes a lot to the variance.

And here is an exchange from an email of May 3 2006:

Also, what was the timing for data collection in the Governorates where the mahdi Army was fighting? Would those deaths be captured in the ILCS total?

This is a bit off the top of my head, but I think not. More generally, the same argument I used for Falujah would be applicable: For these partciular places of intense fighting, it is better to estimate the overall level for the country, and then add the best esitmate for such places that will not be captured.

With regard to the question of ILCS estimates being low, here is an exchange from an October 26 2006 email:

Les Roberts quotes you as saying:
“A survey led by a group in Norway (see report at http://www.fafo.no) estimated 56 violent deaths per day over the first year of occupation, but the authors speculate that the estimate is low.8 ” AND
“Lead Researcher Jon Pederson told Lancet author Richard Garfield that he knows his estimate is low. When revisiting a small sub-sample of household with children, and additional 50% of reported child deaths could be identified. ”
Is this fair as regards the war-related mortality figures? If so, why?, If not, why not?

No it is not, I am frankly rather irriated by Les’ contention that because of the fact that I admit that carrying out surveys in Iraq is difficult, then their work must be much better. In any case – reporting problems on infant mortality and adult mortality are generally quite different. But I do think that we are on the low side.

It’s commonly believed that single items in surveys are less accurate than surveys with at least a section on a topic. You hinted in the Washington Post yesterday that the ILCS mortality could be low for this reason. Could you elaborate? Do you have any evidence one way or another regarding this issue?

yes in general this is true. But what one usually finds is that it is not that bad. It is generally accepoted in the demographic community that household based mortality estimates are on the low side, particularly with respect to mortality where particular (easily know causes) are not speciufied.

With regard to Robert (“Bob”) Shone’s comment in which Pedersen says I may have gotten his position wrong on the Main Street Bias, this is entirely possible, as I was relying upon memory and we had only limited time to discuss. I actually think my summary of his position (Robert, did you ask his opinion on my entire paragraph on the issue, or only on the phrase you extracted, which is taken out of context.) is generally consistent with what he said in the email to Shone.

By the way, Pedersen did NOT think that there was anything to the “Main Street Bias” issue. He agreed, I thought, that, if there was a bias, it might be away from main streets [by picking streets which intersect with main streets]. In any case, he thought such a “bias”, if it had existed, would affect results only 10% or so.

I had not intended to imply that Pedersen didn’t think the Main Street Bias issue was a possible issue for surveys, as we didn’t discuss that. We only discussed whether it could explain the discrepancy between L2006 and the mortality figure that Pedersen believes is correct. The 10% figure came from him spontaneously. He may since have increased his estimate.

In any case, here was his November 28 response to reading my account of his thinking:

No complaints. Wish other people were as gracious.
Thanks,

He doesn’t seem to feel he was grossly misrepresented, as Josh, and, perhaps, Robert S. are suggesting.

First, Donald says:

“So if ILCS undercounted violent deaths and may not have included criminal murders in its range, there’s no reason to claim, as Iraq Body Count did, that there is a conflict.”

First, if any such undercount exists (again I don’t take Soldz’ version as being exactly what Pedersen has said – I suspect Pedersen was less certain about this than in Soldz version) he believes it is small.

Second, IBC already did remove all crime deaths from Lancet before the comparison. The comparison IBC did gave every benefit – undue benefit even – to convergence with Lancet:

1. We did not account for any combatants in ILCS that would be excluded from Lancet (even though Lancet says it would have excluded most military deaths during the invasion – as Stephen does acknowledge in his post above).

2. We did not account for the fact that Lancet subtracted 3,000 violent deaths in the “excess” calculation while ILCS used no such ‘excess’ calculation to subtract pre-war deaths.

3. We did not account for the fact that Lancet excluded Fallujah (which they say represents all of Anbar province). We compared Lancet’s estimate for 97% of Iraq to the ILCS’ estimate for 100% of Iraq. To be truly consistent we should have removed all of the Anbar deaths from ILCS before the comparison. We did not do this either.

All of these would drive the two much further apart than they already are in our comparison. A small underestimate of the kind Pedersen has spoken of would not change things much. And Pedersen saw and approved what we were writing about ILCS before we published it.

Furthermore, Roberts has long emphatically claimed that L1’s non-Falluja estimate is an “extremely conservative” underestimate, and he never ever has claimed that it would likely be only small. So if we are supposed to account for what the authors have conjectured about underestimation in their estimates, this would only, yet again, spread the two much further apart from each other. It would not bring them closer together.

Donald also says:

“IBC has been using ILCS as its justification for claiming that they are picking up at least 50 percent of the violent deaths, but that argument falls apart if Pedersen admits that his survey probably gave an undercount.”

This is also false. So eager to find fault with IBC. Again, Pedersen saw and approved our comparison with ILCS before we published it. He has known all along exactly what we’ve been saying about it. And IBC’s comparison of itself again took a very conservative path and gave every benefit of the doubt against itself. For example, it assumed only 3,000 combatant deaths even including Saddam’s military during the invasion, and assumed that absolutely 0 crime deaths were recorded as war-related deaths by ILCS, removing 100% of these from IBC before the comparison. Both of these assumptions almost certainly understate IBC’s coverage relative to ILCS.

Stephen says:

“Even more relevant is that the ILCS fieldwork in both Fallujah and the Shia south was completed before the fighting. that is, by early April 2004″

Where did you get this from Stephen? ILCS had hundreds of surveyors working up to May 25th. And there is no increase in April 2004 in L1. April is *lower* than March in L1. Only in Falluja is there an April increase. And as I said above, in a truly consistent comparison between ILCS and L1, Falluja should be removed altogether from ILCS just as it was removed from L1.

And I have no idea what you think you’re referring to with “before the fighting” in April 2004 in the Shia south.

One of the things the pro-Lancet crowd has persistently done is spuriously downsize the ILCS timeframe to favor Lancet. Lambert has regularly taken the line that ILCS covered “13 months”, assuming 0% coverage of deaths in May, even as they were surveying up to May 25 2004 (over 14 months after the start of the invasion). That recent piece by Iraqanalysis.org even more ridiculously chopped this down further to “12 months”, excluding all of April and May. There is some leeway here, but the timeframe is neither 13 nor 12 months. It’s somewhere over 13 months but probably somewhat less than a full 14 months.

“In Lancet 2004, something like half the non-Fallujah violent deaths were from April on.”

That’s incorrect too. It’s nowhere near half. Only in the Fallujah cluster is April unusually high. And only July is particularly high in the post-April months outside Falluja. August is very low, and September is low.

“IBC incorrectly used extrapolation on ILCS, when we know the fighting escalated in April.”

Nonsense. Fighting escalated in Falluja in April. But as I said above, Falluja shouldn’t even be included at all in ILCS if we’re doing a comparison to an L1 estimate that is excluding Falluja/Anbar.

Furthermore, IBC applied Lancet 2004’s timeline, so it is increased proportional to Lancet data. Any post-April increase in L2004 would be reflected in our correction of ILCS.

And the same conclusions follow when you don’t do the “extrapolation on ILCS” and go the other way, removing the post ILCS deaths from Lancet:
http://scienceblogs.com/deltoid/2006/04/ibc_takes_on_the_lancet_study.php#comment-82524

No matter how you slice it, the two don’t line up. Regardless, Roberts has recently taken to calling the ILCS a “gross underestimate of deaths”, so I think they’ve abandoned any care about themselves lining up with ILCS.

“By the way, Pedersen did NOT think that there was anything to the “Main Street Bias” issue…”.

I queried this with Jon Pedersen (who hadn’t read the main street bias material prior to his discussion with Stephen). He responded, in an email:

“Yes, probably Stephen Soldz confused the issue somewhat here. There are actual several issues:
1) I very much agree with the MSB-team that there is some main stream bias, and that this is certainly an important problem for many surveys – not only the Iraq Lancet one.
2) I am unsure about how large that problem is in the Iraq case – I find it difficult to separate that problem from a number of other problems in the study. A main street bias of the scale that we are talking about here, is very, very large, and I do not think that it can be the sole culprit.
3) The MSB people have come up with some intriguing analysis of these issues.”
(Jon Pedersen, email to me, 4/12/06)

Mortality estimation using survey instruments has a pretty long history, though it’s only been since perhaps the mid-1960’s that there have been systematic efforts to handle recall bias issues, probably spurred by the explosion in survey data collected from third world countries in that decade. Basically, this is the entire (sub-)field of indirect demographic estimation. Brass’ methods focused on infant and child mortality because they’re the biggest problem — infants who die are typically also more likely to be overlooked in reproductive histories so they are omitted both from the numerator and denominator of rates, and also because infant and child mortality tends to be high and variable. Adults are less often missed (which makes indirect estimation of adult mortality more reliable than for infants and children under five) though there definitely appear to be differentials across characteristics of the decedent and the survey respondent. For example, women tend to be missed more often than men; adult siblings tend to be poorer at recalling events than a parent is; and adult women appear to be better than adult men. That’s why the DHS surveys (demographic and health surveys done in many countries around the world) specify a ranking for preferred respondent.

I emphasize that I do not have any first-hand knowledge of how the JHU/AMU team collected their data, but well-done demographic surveys are designed to help minimize issues of recall bias, much in the way that you’ve described verbal prompting. For example, note that in these two studies, they didn’t just collect counts of deaths — they actually needed household histories in order to calculate person-months of exposure. Household histories are generally regarded as the best way to do this because they tend to have less recall bias. So you’re right, death certificates don’t correct for omissions; but household histories minimize omissions and then the DC can be used to hammer down the timing. As an aside, the ILCS/IMIRA questionnaire didn’t ask its mortality question in this way so there’s no “timing” in the reported deaths. Likewise, many crisis mortality studies don’t collect household histories this way, which is probably why the SMART guidelines are for short recall periods.

As for the bootstrap CIs, Pedersen was obviously talking about parametric CIs. In both the Roberts and Burnham papers, bootstrapping is mentioned at the bottom of (their respective) page 3. I sort of disagree that the bootstrap is tricky — it’s impressively simple-minded (though it took a very bright guy like Brad Efron to figure out something so simple). In any event, there is a big issue about whether it makes sense to bootstrap the cluster means, and how one gets from relative risk ratios to a point estimate of total excess deaths, but that isn’t strictly about the bootstrap procedure itself.