November 26th, 2006
I have posted the following comments on the Media Lens Message Board as part of a discussion of Jon Pedersen’s skepticism regaring the Lancet 2006 Iraq mortality figures.
I have had a number of emails with Jon Pedersen going back to last spring, if I remember correctly. I also had the privilege of having lunch with him and discussing these issues for almost a couple of hours.
He struck me as extremely thoughtful and as having no ax to grind. As I relate his thoughts, [I am responsible for this account, which could be incorrect on details. Comments in brackets are my thoughts.] he believes the Lancet rates are too high. He thinks two things are involved, and had two other comments on the methodology.
1. The prewar mortality is too low. This would be due to recall issues. He asserted that demographers are well aware that there are problems with assessing mortality by survey more than 1 year in the past. [See, for example, the SMART methodology which states that one should never assess mortality beyond one year: "Accuracy: The shorter the recall period, the more accurate the estimate of mortality. This is because more distant events are more likely to be forgotten by respondents and there are likely to be more mistakes in the time of death. (Recall periods longer than one year should not be used.)" (p. 31-32)]
[As a research psychologist, this is not surprising. We routinely find that memory, even for major events, gets transformed relatively quickly. As one minor example, one half of those who, at age 18 reported major depression did not report ever having had major depression ever, when reinterviewed at age 21. Especially in the context of a brief interview, it is entirely possible that people could briefly "forget" deaths a ways back, or "choose" not to talk about them as they may be painful. I, too, was concerned about this recall issue.] Pedersen said it was possible that this effect may be larger for nonviolent than for violent deaths.
[Note that, if the UN prewar mortality rate of 9.03 is correct, this will not, by itself, change the qualitative message of the study. My calculations are that the excess mortality would then be about 350,000. But other problems would arise, such as the mortality rates for the two years postwar, which would then be low. These might support the Pedersen/SMART position that 1 year prior is the outer limit for good results with this methodology.]
2. Pedersen thought that people were likely reporting nonviolent deaths as violent ones. [One argument for this is the surprisingly low postwar mortality rates in the Lancet 2006 study, which Pederesen thought were unlikely to be correct.] We did not discuss the precise mechanisms for this misreporting, or Pedersen’s other reasons for believing it.
[We did discuss the issue of the death certificates, to which Pedersen said "who knows what's on them." I have since found two accounts of death certificates which indicated that they have details of the death. More details from the Lancet authors as to the details of how and to what degree interviewers examined the certificates would be helpful.]
3.A third issue concerns the inability, because of security, of the US Lancet team to supervise the interviewers. Pedersen raised the possibility of his FAFO team repeating the survey, but was extremely hesitant as they would not be able to closely supervise the interviewers and,therefore, could not vouch for the protocol being followed exactly.
4. Pedersen believes the confidence intervals (CIs) for the Lancet studies are too small. This involves two issues. First, the calculated CIs only include sampling error, but the study also includes nonsampling error, as in when a different household was chosen for security reasons or the protocol wasn’t followed for other reasons. Recall issues and other forms of inaccurate reporting would also increase nonsampling bias.
A second reason the CIs are too small is a technical one. Death is a relatively rare event. The statistical methods used to calculate CIs [linearized variance estimators] are based on an assumption that the events aren’t too rare. Thus, the CIs are likely too small.
[While the Lancet 2006 authors say they also calculated bootstrap standard errors, they give no detail on how this is done, and are not clear which CIs they report. Bootstrap CIs with survey data are very complicated and can be done many different ways with, potentially, different results.]
Two additional points: Pedersen acknowledges that the ILCS cannot be the gold standard for mortality as asking one brief set of questions at the end of a long survey would lead to underreporting of mortality. Thus, he assumes that the ILCS figures are low. But he thinks they are in the ballpark.
Another point: He thinks that the IBC argument about how the number of bombings reported in the Lancet 2006 study would result in such a large number of wounded that it seems too large. If I understood correctly, he does not buy the IBC argument about the inconsistency between the estimated wounded and those reported as treated in hospitals, as he agreed [I thought] with a point I made that there were likely serious problems with Iraq’s health surveillance systems, so that these figures cannot be taken very seriously. But, he felt that 800,000 or more wounded would swamp any public heath system in the world. [An additional point occurred to be after the conversation. As one who has read the press on Iraq, including Iraqi bloggers, very carefully, I can't remember reading articles about the walking maimed of Iraq. It does seem that, if the Lancet mortality figures were correct, that the magnitude of wounded would be such as to be reported frequently.]
I want to reiterate that Jon Pedersen, as far as I could tell, was extremely thoughtful. Several times he raised an argument only to raise a counterargument. He did NOT appear to be argumentative or to be involved in proving a point. I was extremely impressed. Therefore, I take his arguments and thoughts seriously. He may know more about middle east demography and survey methodology than anyone, so his judgments and intuitions cannot be simply dismissed.
I would also remind people that Les Roberts refers to Pedersen as “he highly revered Jon Pederson.” Pedersen also spoke highly of Roberts. These are scientific debates, and scientific debates do not always have bad guys. They also are often unresolvable with the current state of knowledge.
By the way, Pedersen did NOT think that there was anything to the “Main Street Bias” issue. He agreed, I thought, that, if there was a bias, it might be away from main streets [by picking streets which intersect with main streets]. In any case, he thought such a “bias”, if it had existed, would affect results only 10% or so.
As to his not communicating with Les Roberts, both he and I have sent emails to Les [I also sent one to Gilbert Burnham] which have not been responded to. There have been hundreds of emails on this issue, and people are very busy. Emails have been missed by many participants. Please cut out the snide comments about why people didn’t send emails to someone.
As to Pedersen’s writing a letter on this, I would hope he would. On the other hand, remember that he’s got a lot of other things going on. As far as I’m aware, he hasn’t sought out the role of critic but simply responded to those who asked his opinion. While such a communication would be very helpful, he is not obligated to undertake it.
So, the bottom line is that two experts differ. The rest of us are left trying to assess the differences. But we are also left not knowing.
That said, I’m not sure that the true mortality figure is as low as the 150,000 that Pedersen cites. I suspect the true figure is somehat higher, but below the 650,000 of Lancet 2006. But I can’t provide any strong arguments for my instincts here. I now take 150,000 that as the rock bottom number for Iraqi excess deaths. There may be more, even many more, but we can’t be sure. It simply is very difficult to get accurate figures in a time of chaos.
I would say that the moral case regarding the war is well made if the true excess mortality is “only” 150,000. This war and occupation has resulted in horrifying death and destruction. It is morally unjustifiable on any grounds of reducing harm.