Posts filed under 'Research Methods'

Tirman: Implications of Iraq mortality studies

John Tirman, the director of MIT’s Center for International Studies, which funded the second Lancet Iraq mortality study, has an Op Ed in the Boston Globe this weekend on the implications of the multiple Iraq mortality studies. [One assumes that this is, in part at least, a response to the recent despicable and dishonest right wing hatchet jobs on the Lancet 2 study in the National Review, and by Globe columnist Jeff Jacoby]:

The murky toll of the Iraq war

By John Tirman
January 19, 2008

ONCE AGAIN, a controversy has erupted over how many people are being killed in Iraq. It’s an important debate, not only for beleaguered Iraqis, but for Americans seeking stability and a timely exit.

Mortality figures alone can tell a compelling story. Add to that other numbers that fill in our understanding even more - such as the scale of the flow of refugees or the women widowed by the war - and we have useful information.

So what are these statistics, and what do they tell us about this nearly five-year-old conflict?

Two kinds of accounts have emerged on the question of mortality. One is a literal count, body by body, from reports in the English language press. Because the media, mostly based in Baghdad, cannot grasp most of the violence, this is an undercount (now about 84,000) even by the reckoning of its authors, the UK-based Iraq Body Count.

The second method is to go out and ask the question in surveys of randomly selected households. This has been done five times under very dangerous conditions. Surveys of this kind during war are relatively new, and, as a result, it’s not surprising that the numbers they’ve produced have varied. But there is significant congruence.

The surveys agree that mortality is much higher than is typically held in political discussions about Iraq. The highest figure, from Opinion Business Research, a private survey firm in London, is 1.2 million through August 2007. It is also the most recent.

About 15 months ago, a survey commissioned by my center at MIT and published in The Lancet found that 601,000 had died by violence through June 2006. This figure has created a firestorm of criticism, but the methods are sound and none of the many peer reviews found anything greatly amiss. (One recalculation brought the death-by-violence total down to 450,000.)

Then last week, Iraq’s Ministry of Health released its large survey, also ending in June 2006, finding that 151,000 had died by violence. But their data tables show an enormous “excess death” total of nearly 400,000 caused by the war, and a peculiarly flat rate of violence throughout the war. Because the interviewers worked for the government, it’s likely that many respondents attributed deaths to nonviolent causes, in order to protect themselves from unwanted attention.

What to make of all this? The first conclusion is that hundreds of thousands of people have died as a result of the war - this seems incontrovertible. It is buttressed by the large number of displaced - some 3 million to 3.5 million caused by the war - and a reported total of 500,000 war widows.

The second conclusion, which helps us understand the violence, is that such a human catastrophe accounts for the insurgency in ways that no other explanation does. Whatever one makes of these insurgents, they appear to be fighting to defend their towns and tribes (apart from Al Qaeda’s foreign operation). Violence begets violence, especially when foreigners are involved.

The third conclusion is that Iraq’s devastation runs deep and wide. A generation of young men is being wiped out. Many of the most educated have left. The poverty of widespread widowhood may become chronic. The healthcare system is in shambles. Neighborhoods and towns ethnically cleansed means long-lasting displacement for tens of thousands. The humanitarian aid challenge is vast, and will last for many years.

How this affects US strategy is complex, of course, but two things stand out. First is that strategies to reduce violence against civilians and to increase economic and physical security are paramount. US leaders seem to grasp this, but their actions (arming Sunni militias, for example) may prove foolhardy.

Second, Iraq’s neighbors must be part of the solution, given the scale of misery. President Bush has never embraced this idea, but it seems more and more obvious as the war drags on. Yet on Bush’s recent trip to the region, Iraq was nearly absent from his agenda.

The lessons from the killing fields and refugees and widows won’t go away. The sooner we fully realize the scale of this catastrophe, the better we may be able to work on reconstructive remedies.

John Tirman is executive director and a principal research scientist at MIT’s Center for International Studies.

Add comment January 20th, 2008

Comments on the reception of new NEJM Iraq mortality study

I have just posted the following comments on the new NEJM study of Iraq mortality on the Media Lens Message Board, in response to heated criticism of the new study:

 I don’t think this is fair. The NEJM study is another attempt to do something very difficult: assess the consequences of the war and occupation in a situation of extreme violence. I notice that Les Roberts was fairly positive, while raising a number of important issues. [One version of Les' thoughts can be read  [url=http://psychoanalystsopposewar.org/blog/2008/01/10/les-roberts-on-new-iraq-mortality-study/]here[/url]] I think we should follow Les’ example here.

While there are many issues with the new study, there is no fatal flaw.

I think the authors assess deaths due to violence because they believe, rightly or wrongly, that they can estimate this figure more accurately than total excess mortality. Violent deaths do not rely upon an accurate estimate of prewar mortality several years earlier, while excess mortality estimates do. This is something in which the NEJM study clearly fails. This failure does suggest, as  Les suggests, that the NEJM study is an undercount. Other problems are, as Cockburn points out, the steady rate of mortality in the NEJM study.

But Cockburn attributes nefarious motivations to the fact that Iraqi interviewers were sent to Amman for training. I will bet that this was so that they could be trained by WHO staff. Remember, at this time Les Roberts and, I believe, Gilbert Burnham, went to Amman when conducting L2 and conducted the data analyses there. The reason was the same: they felt it was too dangerous for foreigners to go into Iraq.

As for the ORB study, I was impressed when it came out. but the absence of any publication of methodological details, much less their failure to post the additional results they promised for early October cast doubt upon the study. Until they publish more, it can’t be taken as meaning much of anything, alas.

I’m afraid we’re in danger of falling into a dangerous trap of defending heartily studies whose the results we like and attacking those whose results we dislike. I teach my research students that we should subject studies confirming our prior beliefs to extra scrutiny while being careful not to search mightily for methodological flaws in those studies we don’t agree with. Otherwise, we learn nothing.

If Les welcomes this study, while examining its weaknesses, I suggest we should as well. Examining violent mortality in Iraq is extremely difficult. we may never know what the true figure is. As of summer 2006, it was most likely somewhere between 150,00 and 650,000. By now, it is probably somewhere between 250,00 and 1.2 million.  In any terms, that is truly horrifying and a humanitarian catastrophe. We should work to get that message out. To fight NEJM vs Lancet will only deflect the  message and work to the right’s advantage. Let’s not give them that advantage.

I suspect there will be responses at the Media Lens Message Board. Go there and read them.

Add comment January 12th, 2008

Les Roberts on new Iraq mortality study

Les Roberts, an author of the two previous Lancet studies of Iraq mortality,  sends the following comments on the new Iraq mortality study, the Iraq Family Health Survey, that I blogged about last night:

I think that this new article in the NEJM is a good addition to the discussion. It is good for Iraqis, it is good for science, it is good for promoting peace.

1) There is far more in common in the results than appears at first glance.

The NEJM article found a doubling of mortality after the invasion, we found a 2.4 fold increase. They found a CMR of 3/1000/yr. before and 6 after but thought they were missing almost 1/2 the deaths. We found a CMR of 5 before and 13 after….thus we actually agree roughly on the number of excess deaths. The big difference is that we found almost all the increase from violence, they found 1/3 the increase from violence.

The other odd items (family size, refusal rates, absentee household rates, fraction of deaths from infectous diseases and car accidents…) are strikingly similar.

IBC adds to their estimate for months after a given date; back at the end of June 2006, IBC estimated 41,000 deaths (my notes suggest 38,475 to 42,889 on June 24, 2006). This new estimate is 4 times the “widely accepted” number of that moment, our estimate was 12 times higher. Both studies suggest things are far worse than our leaders have reported.

2) There are reasons to suspect that the NEJM data had an under-reporting of violent deaths.

The death rate they recorded for before the invasion (and after) was very low….lower than neighboring countries and 1/3 of what WHO said the death rate was for Iraq back in 2002.

The last time this group (COSIT) did a mortality survey like this they also found a very low crude death rate and when they revisited the exact same homes a second time and just asked about child deaths, they recorded almost twice as many. Thus, the past record suggests people do not want to report deaths to these government employees.

We confirmed our deaths with death certificates, they did not. As the NEJM study’s interviewers worked for one side in this conflict, it is likely that people would be unwilling to admit violent deaths to the study workers.

They roughly found a steady rate of violence from 2003 - 2006. Baghdad morgue data, Najaf burial data, Pentagon attack data, and our data all show a dramatic increase over 2005 and 2006.

Finally, their data suggests 1/6th of deaths over the occupation through 6/06 were from violence. Our data suggest a majority of deaths were from violence. All graveyard reports I have heard are consistent with our results.

I hope these comments are helpful. I hope people in the press will visit a few  graveyards/morgues/hospitals and decide if 1/6th or over 1/2 of the deaths during the period 2003-06.

Best regards,

Les Roberts

5 comments January 10th, 2008

New estimate of Iraq violent mortality, 151,000, far lower than Lancet

A new household survey in Iraq released today comes up with an estimate for violent mortality far lower than that by Burnham, Roberts and others in the Lancet last year. they estimate 151, 000 violent deaths. The study is available online from the New England Journal of Medicine. [I have not read the study yet but felt that I should post it as soon as possible.] Here is an article from MedPage Today on the study:

Violence-Related Mortality in Iraq from 2002 to 2006

By Peggy Peck

GENEVA, Jan. 9 — The latest audit of Iraq war deaths — combatants and civilians — puts the mortality toll at 151,000, which is about 75% lower than an estimate published more than a year ago.

The new estimate comes from the Iraq Family Health Survey Study Group, a joint project of the World Health Organization and the Iraqi government that did interviews in 9,345 households in about 1,000 locations across Iraq. The findings were published online today by the New England Journal of Medicine.

The total mortality was based on direct reporting of deaths by household respondents, and the authors said that “there are no better methods available to provide more accurate estimates of the death toll due to the humanitarian conflict in Iraq in the wake of the 2003 invasion.”

The Iraq Family Health Study was a representative study of 9,345 households that have been providing information on deaths since 2001. The new estimate was culled from surveys conducted in 2006 and 2007 with 1,086 household clusters that reported 1,325 deaths from January 2002 through June 2006.

But estimating deaths in a war zone is not an exact science and the authors said that when “underreporting was taken into account, the rate of violence-related death was 1.67 (95% uncertainty range: 1.24 to 2.30). This rate translates into an estimated number of violent deaths of 151,000 (95% uncertainty range: 104,000 to 223,000) from March 2003 through June 2006.”

In other words, the number of violent deaths could be as low as 104,000 or as high as 223,000 during those three years, but the most probable total was 151,000, said Mohamed Ali, Ph.D., a WHO statistician who was a member of the Iraq Family health Survey Study Group.

And even the high-end estimate of this survey was about 66% lower than the mortality estimate from Burnham et al (Lancet 2006; 368: 1421-8), which estimated that there were 601,027 violent deaths during the three years following the U.S. invasion of Iraq in March 2003.

But the Iraq Family Health Survey estimate was about three times higher than the deaths reported by the Iraq Body Count project, which reported 47,668 deaths.

“How is it that these numbers vary so widely, given that there can only be one true answer?” asked Catherine A. Brownstein, M.P.H., of Yale, and John S. Brownstein, Ph.D., of Harvard in an editorial that accompanied the survey results.

“The [Iraq Family Health Survey] study group does not directly address this question, but it deserves speculation,” the Brownsteins added. “The probable cause is that the techniques used to obtain the estimates differ radically from one another.”

The Iraq Body Count project crosschecks civilian deaths that are reported by a minimum of two independent sources against hospital and morgue records, as well as official figures. Moreover, it does not include deaths of combatants in its tally. It represents surveillance, not survey, and “should be treated as a reliable lower bound,” according to the editorialists.

The Burnham study used a method similar to that of the Iraq Family Health Survey but its overall sample involved only 1,849 households in 47 cluster areas, while the new survey included almost five times as many households and 20 times more regions or clusters.

And although all three surveys identify the same regions as high mortality areas — Anba, Babylon, Basra, Diyala, Nineveh, and Salahuddin — there were striking differences in the death rates in those areas reported by Burnham et al versus the Iraq Family Health Survey.

According to Burnham et al, the daily death rates in those regions for years 2003 to 2006 were 231, 491, and 925, respectively, versus 128, 115, and 126 for the same periods in the Iraq Family Health Survey.

All three surveys also agreed that Kurdistan was a low mortality area and Baghdad was a high mortality area. But although 54% of the deaths reported in the Iraq Family Survey and 60% of the deaths reported by the Body Count project occurred in Baghdad, only 26% of the deaths reported by Burnham et al occurred in Baghdad.

Finally, the authors concluded that even though their estimate was “substantially lower than that estimated by Burnham et al, it nonetheless points to a massive death toll in the wake of the 2003 invasion — and represents only one of the many health and human consequences of an ongoing humanitarian crisis.”

The survey was funded by the United Nations Development Group Iraq Trust Fund, the European Commission, and the WHO.

The study group members reported no potential conflicts of interest.

Here is the Abstract from the paper:

Violence-Related Mortality in Iraq from 2002 to 2006

Iraq Family Health Survey Study Group

ABSTRACT

Background Estimates of the death toll in Iraq from the time of the U.S.-led invasion in March 2003 until June 2006 have ranged from 47,668 (from the Iraq Body Count) to 601,027 (from a national survey). Results from the Iraq Family Health Survey (IFHS), which was conducted in 2006 and 2007, provide new evidence on mortality in Iraq.

Methods
The IFHS is a nationally representative survey of 9345 households that collected information on deaths in the household since June 2001. We used multiple methods for estimating the level of underreporting and compared reported rates of death with those from other sources.

Results
Interviewers visited 89.4% of 1086 household clusters during the study period; the household response rate was 96.2%. From January 2002 through June 2006, there were 1325 reported deaths. After adjustment for missing clusters, the overall rate of death per 1000 person-years was 5.31 (95% confidence interval [CI], 4.89 to 5.77); the estimated rate of violence-related death was 1.09 (95% CI, 0.81 to 1.50). When underreporting was taken into account, the rate of violence-related death was estimated to be 1.67 (95% uncertainty range, 1.24 to 2.30). This rate translates into an estimated number of violent deaths of 151,000 (95% uncertainty range, 104,000 to 223,000) from March 2003 through June 2006.

Conclusions Violence is a leading cause of death for Iraqi adults and was the main cause of death in men between the ages of 15 and 59 years during the first 3 years after the 2003 invasion. Although the estimated range is substantially lower than a recent survey-based estimate, it nonetheless points to a massive death toll, only one of the many health and human consequences of an ongoing humanitarian crisis.

More information on the Iraq Family Health Survey is available from their web site.

2 comments January 9th, 2008

Are teen sex and delinquenct linked?

The Washington Post has an article partially on a study claiming to debunk the finding that early teen consensual sex leads to later delinquency. By comparing pairs of twins, they suggest that the effect might even be in the opposite direction. While hardly definitive, it looks like a nice attempt to use sophisticated data analysis along with careful conceptual analysis to clarify important social issues.

The article also discusses other issues,including a claim that the relationship between IQ and breastfeeding, about which I blogged last week, may be artifactual:

A recent study by Scottish researchers asked whether the higher IQs seen in breast-fed children are the result of the breast milk they got or some other factor. By comparing the IQs of sibling pairs in which one was breast-fed and the other not, it found that breast milk is irrelevant to IQ and that the mother’s IQ explains both the decision to breast-feed and her children’s IQ.

I suspect neither of these studies is the last word on the breastfeeding-IQ link.

Another issue involves the benefits and risks of early parenting by poor black girls. For decades, Arline Geronimus has made the case that poor young girls are often better off having children young, in direct contradiction of our society’s obsession with preventing teen pregnancy. I was in a seminar with Arlene during our postdocs in 1986-87 and remember her making a quite plausible case. She apparently has continued to amass evidence over the 20 years since.

While none of these studies are definitive, they all emphasize the importance of careful research work before developing social interventions to address what, in some instances, are nonexistent problems.

Here’s the entire article:

Study Debunks Theory On Teen Sex, Delinquency
New Analyses Challenging Many Old Assump
tions

By Rick Weiss

Researchers at Ohio State University garnered little attention in February when they found that youngsters who lose their virginity earlier than their peers are more likely to become juvenile delinquents. So obvious and well established was the contribution of early sex to later delinquency that the idea was already part of the required curriculum for federal “abstinence only” programs.

There was just one problem: It is probably not true. Other things being equal, a more probing study has found, youngsters who have consensual sex in their early-teen or even preteen years are, if anything, less likely to engage in delinquent behavior later on.

That new analysis, a reworking of the same data the Ohio team used, is one of several recent instances in which a more precise parsing of data has begun to turn long-standing societal presumptions on their head. By bringing evidence to bear on complex social issues, these studies are forcing individuals and policymakers to rethink such hot-button topics as the benefits of breast-feeding, the risks of teen child-bearing and, in the latest example, the harms long presumed to result from teen sex.

Like many of the newer studies, the latest one — led by Paige Harden, a doctoral candidate in psychology at the University of Virginia in Charlottesville — used the powerful techniques of behavioral genetics. The field specializes in studies on twins, research that can help tell whether behavioral traits are the result of genes or the social environment, and that has periodically stirred controversy when it has focused on the genetic underpinnings of criminality and intelligence.

But the specialty’s analytic methods can also help tell whether one behavior, such as early sex, is merely correlated with or actually causes a second behavior that is often found with it, such as delinquency. If two behaviors often exist in the same people but are found not to be connected by cause and effect, then a third factor is likely to be causing both.

That kind of finding can help identify better targets for prevention efforts, experts say.

“Behavioral geneticists have long sought to establish causal links between genes and complex behaviors. So it’s fascinating to see them use the tools of their trade to dispute widely held beliefs” about the social roots of some of those behaviors, said Erik Parens, a senior research scholar who has tracked the field intensively at the Hastings Center, a Garrison, N.Y., science and ethics think tank.

The latest example started when Dana Haynie, a sociologist at Ohio State, and her then-graduate student, Stacy Armour, published a study in February in the Journal of Youth and Adolescence. They analyzed data collected from more than 7,000 children as part of the National Longitudinal Study of Adolescent Health, a federally funded survey that in 1994 began gathering information about the health-related behavior of U.S. schoolchildren who were then in grades seven through 12.

Haynie and Armour divided the children into three groups based on when they first had sex: when they were younger, about the same age or older than the age at which most of their local peers lost their virginity. (It varies by region, but on average, U.S. children lose their virginity at age 16.) They also compiled information on graffiti-painting, shoplifting, drug-selling and other “problem behaviors” by those young people in later years.

Their conclusion: One year after losing their virginity, children in the early category were 20 percent more likely than those who started having sex at the average age to engage in delinquent behavior, even when several other relevant factors such as wealth, race, parental involvement and physical development were taken into account.

Those findings supported the widely held notion that loss of virginity at a relatively young age appears to, as Haynie and Armour wrote, “open the doorway to problem behaviors.”

Harden, at the University of Virginia, didn’t believe it.

Looked at from a similarly high altitude, she said, people might conclude that red meat is a health food, since people live longer in countries where more is eaten. Only when the issue is studied within one country does red meat’s link to chronic diseases appear.

Suspecting such an error in the Haynie study, Harden and three colleagues, including her adviser, Eric Turkheimer, an expert in behavioral genetics, studied more than 500 pairs of twins in the same national survey analyzed by the Ohio team. Because twin pairs share similar or identical genetic inheritances (depending on whether they are fraternal or identical) and the same home environment, twin studies are useful for seeing through false cause-and-effect relationships.

The team looked at identical twin pairs in which one twin initiated sex younger than the other, then team members tallied subsequent problem behaviors. If sex really adds to the chances of delinquency, then early-sex teens should end up delinquent more often than their later-sex twins.

“It turns out that there was no positive relationship between age of first sex and delinquency,” Harden said.

The way to reconcile that with the previous evidence of a link is to conclude that some other factors are promoting both early sex and delinquency, she said. In an e-mail, Haynie agreed. And the Virginia study, to appear in the March 2008 issue of the Journal of Youth and Adolescence, offers some clues.

It found that identical twins, who have the same DNA, were more similar to one another in the ages at which they lost their virginity than were fraternal twins, whose DNA patterns are 50 percent the same — an indication that genes influence the age at which a person will first have sex. Other twin studies have found the same pattern for delinquency.

Together, those findings suggest that some genes — perhaps, for example, those that increase impulsivity and risk-taking — may underlie both behaviors.

“You need to have some appetite for risk-taking to be a delinquent. And the same if you’re 13 and going to have sex for the first time,” Harden said.

Efforts to prevent delinquency can hardly take aim at people’s genes. But the Virginia study also indicates that social factors, as yet unidentified but perhaps involving relationships with family and friends, have an even bigger impact than genes on whether a child will become delinquent. Those are the things that should be identified and targeted by delinquency-prevention programs, said Jeanne Brooks-Gunn, co-director of Columbia University’s National Center for Children and Families.

“I wouldn’t be focusing on early sexuality . . . to alter rates of delinquency,” she said.

Perhaps most surprising, the Virginia study found that adolescents who had sex at younger ages were less likely to end up delinquent than those who lost their virginity later. Many factors play into a person’s readiness for sex, but in at least some cases sexual relationships may offer an alternative to trouble, the researchers say.

Even then, there are emotional and physical risks. Young adolescents, in particular, are less likely to use condoms and so are vulnerable to sexually transmitted diseases and unwanted pregnancies.

But those are risks that other nations have mitigated with education, Harden and Turkheimer said, while U.S. educators wanting a piece of the nation’s $200 million “abstinence only” budget must adhere to a curriculum that links sex to delinquency and explicitly precludes discussion of contraception.

The new study “really calls into question the usefulness of abstinence education for preventing behavior problems,” Harden said, “and questions the bigger underlying assumption that all adolescent sex is always bad.”

Similar re-analyses have begun to undermine other conventional notions about health.

A recent study by Scottish researchers asked whether the higher IQs seen in breast-fed children are the result of the breast milk they got or some other factor. By comparing the IQs of sibling pairs in which one was breast-fed and the other not, it found that breast milk is irrelevant to IQ and that the mother’s IQ explains both the decision to breast-feed and her children’s IQ.

In another example, Arline Geronimus, a University of Michigan professor of health behavior who is now a fellow at Stanford University’s Center for Advanced Study, knew that babies born to teenagers are more likely to die in their first year of life than those born to older women.

“But that is an apples-to-oranges comparison,” she said. In New York City, for example, far more teen mothers live in Harlem than on the Upper East Side, she said, and “there are a lot of differences between those groups.”

So Geronimus looked more closely and got a different answer.

“If you compare Harlem teen moms to Harlem older moms, you find that the kids of the teen moms are actually less likely to die,” she said. The reasons include the fact that, unlike older women, poor teenagers are generally not juggling jobs and have older relatives to help.

It can make sense for poor women to have children when they are quite young, Geronimus concludes, and any effort to change that ought to treat it as an economic problem, not a health education problem.

In a different re-analysis, Geronimus made another counterintuitive finding. While it is true that, in general, teen mothers are less likely to breast-feed their babies than older moms, it is not true among poor women. Poor teenagers are actually more likely to breast-feed than poor older moms, in large part because the older women have jobs that don’t grant them the time to breast-feed or pump milk.

Because of that misconception, programs promoting breast-feeding have targeted teens instead of older women, Geronimus said. And they have taken aim, in part, at a concern that teenagers were believed to have: the cosmetic effects of breast-feeding on their breasts.

“So you’ve targeted the wrong population,” Geronimus said, “and come up with the wrong kind of intervention.”

1 comment November 11th, 2007

ORB Iraq mortality poll to be reexamined

The British polling company, ORB, that reported that an estimated 1.2 million Iraqis had died from violence post-invasion has decided that their results might be affected by a sampling design that undersampled rural areas. They had taken the survey results off the front page of the Newsroon page for a while. Now it has returned to that page, with a press release announcing that they are conducting some additional surveys in rural areas.

Iraq Casualties Poll Update

ORB to conduct additional interviews in rural Iraq.

We have received a lot of interest in the recent ‘casualties’ poll that we carried out in conjunction with our local partners in Iraq - IIACSS. Our survey was carried out throughout Iraq and looked at the incidence of civilian deaths since 2003 - together with Iraqi’s views on the displacement of people.

As with many other activities in Iraq polling has its own restraints and it is simply too dangerous for interviewers to operate in some areas. Meanwhile local authorities prevent interviewers from working in certain towns and districts. This means that we cannot gather opinion from the more volatile areas but, at the same time we have, so far, also limited coverage in rural districts. Both of these factors mean that any estimate of deaths will remain just that - an estimate.

While, for obvious reasons, we cannot boost our representation of people living in Iraq’s most violent areas we have decided - following feedback from readers of our poll - to conduct a more extensive survey of rural areas to see how this may impact on our estimate. We are in the process of conducting additional interviews in rural areas of Iraq. Once this data has been verified and merged with our current data set we will post it here on the ORB.

As scientists, we await the release of these additional results. I also hope that ORB will post details on their sampling methodology and survey administration techniques. Like the Johns Hopkins studies, the ILCS, and others, we need to closely examine their methodology. While I take the Hopkins and ORB studies seriously as evidence that the death toll is in the many hundreds of thousands, as a scientist, and sometime survey researcher (albeit in a very different area), I know that estimates in a situation like Iraq are just that, estimates. I’d like to see their methodology to see how well they dealt with the various potential biases. If it stands up, the replication of the basic findings of the Hopkins study, that the mortality in Iraq is enormous, will immeasurably strengthen our case.
At the same time, as I teach my research methods students, we should be skeptical of studies that support our prior beliefs and subject them to the same critical examination that we do of studies which challenge our prior beliefs.

1 comment September 21st, 2007

Senator Craig scandal in the light of social research

The New York Times opinion Section has an article by Laura MacDonald that shed new light on the Sen. Larry Craig bathroom sex case. She draws upon the 1970 research of Laud Humphreys on sexual behavior in men’s bathrooms. Humphreys showed that men’s sexual advances in bathrooms are elaborately ritualized, so that, the author argues, the policeman had to have responded to Craig and he must have been entrapped.

The Humphreys study is described in the social research methods text I use for my undergraduate class, put only in terms of the ethical issues raised. Its nice to see it entering popular consciousness for its findings. The article also illustrates that research ca provide illumination for important social questions.

The article:

America’s Toe-Tapping Menace

by Laura MacDonald

WHAT is shocking about Senator Larry Craig’s bathroom arrest is not what he may have been doing tapping his shoe in that stall, but that Minnesotans are still paying policemen to tap back. For almost 40 years most police departments have been aware of something that still escapes the general public: men who troll for sex in public places, gay or “not gay,” are, for the most part, upstanding citizens. Arresting them costs a lot and accomplishes little.

In 1970, Laud Humphreys published the groundbreaking dissertation he wrote as a doctoral candidate at Washington University called “Tearoom Trade: Impersonal Sex in Public Places.” Because of his unorthodox methods — he did not get his subjects’ consent, he tracked down names and addresses through license plate numbers, he interviewed the men in their homes in disguise and under false pretenses — “Tearoom Trade” is now taught as a primary example of unethical social research.

That said, what results! In minute, choreographic detail, Mr. Humphreys (who died in 1988) illustrated that various signals — the foot tapping, the hand waving and the body positioning — are all parts of a delicate ritual of call and answer, an elaborate series of codes that require the proper response for the initiator to continue. Put simply, a straight man would be left alone after that first tap or cough or look went unanswered.

Why? The initiator does not want to be beaten up or arrested or chased by teenagers, so he engages in safeguards to ensure that any physical advance will be reciprocated. As Mr. Humphreys put it, “because of cautions built into the strategies of these encounters, no man need fear being molested in such facilities.”

Mr. Humphreys’s aim was not just academic: he was trying to illustrate to the public and the police that straight men would not be harassed in these bathrooms. His findings would seem to suggest the implausibility not only of Senator Craig’s denial — that it was all a misunderstanding — but also of the policeman’s assertion that he was a passive participant. If the code was being followed, it is likely that both men would have to have been acting consciously for the signals to continue.

Mr. Humphreys broke down these transactions into phases, which are remarkably similar to the description of Senator Craig’s behavior given by the police. First is the approach: Mr. Craig allegedly peeks into the stall. Then comes positioning: he takes the stall next to the policeman. Signaling: Senator Craig allegedly taps his foot and touches it to the officer’s shoe, which was positioned close to the divider, then slides his hand along the bottom of the stall. There are more phases in Mr. Humphreys’s full lexicon — maneuvering, contracting, foreplay and payoff — but Mr. Craig was arrested after the officer presumed he had “signaled.”

Clearly, whatever Mr. Craig’s intentions, the police entrapped him. If the police officer hadn’t met his stare, answered that tap or done something overt, there would be no news story. On this point, Mr. Humphreys was adamant and explicit: “On the basis of extensive and systematic observation, I doubt the veracity of any person (detective or otherwise) who claims to have been ‘molested’ in such a setting without first having ‘given his consent.’ ”

As for those who feel that a family man and a conservative senator would be unlikely to engage in such acts, Mr. Humphreys’s research says otherwise. As a former Episcopal priest and closeted gay man himself, he was surprised when he interviewed his subjects to learn that most of them were married; their houses were just a little bit nicer than most, their yards better kept. They were well educated, worked longer hours, tended to be active in the church and the community but, unexpectedly, were usually politically and socially conservative, and quite vocal about it.

In other words, not only did these men have nice families, they had nice families who seemed to believe what the fathers loudly preached about the sanctity of marriage. Mr. Humphreys called this paradox “the breastplate of righteousness.” The more a man had to lose by having a secret life, the more he acquired the trappings of respectability: “His armor has a particularly shiny quality, a refulgence, which tends to blind the audience to certain of his practices. To others in his everyday world, he is not only normal but righteous — an exemplar of good behavior and right thinking.”

Mr. Humphreys even anticipated the vehement denials of men who are outed: “The secret offender may well believe he is more righteous than the next man, hence his shock and outrage, his disbelieving indignation, when he is discovered and discredited.”

This last sentence brings to mind the hollow refutations of figures at the center of many recent public sex scandals, heterosexual and homosexual, notably Representative Mark Foley, the Rev. Ted Haggard, Senator David Vitter and now Senator Craig. The difference is that Larry Craig was arrested.

Public sex is certainly a public nuisance, but criminalizing consensual acts does not help. “The only harmful effects of these encounters, either direct or indirect, result from police activity,” Mr. Humphreys wrote. “Blackmail, payoffs, the destruction of reputations and families, all result from police intervention in the tearoom scene.” What community can afford to lose good citizens?

And for our part, let’s stop being so surprised when we discover that our public figures have their own complex sex lives, and start being more suspicious when they self-righteously denounce the sex lives of others.

Laura M. Mac Donald is the author of “The Curse of the Narrows: The Story of the 1917 Halifax Explosion.”

Add comment September 2nd, 2007

Newstudy: Ritalin use doubles after divorce

A very interesting study. Unusually, the reporter accurately conveys the complexity of interpreting non-experimental data. The study suggests that divorce may be an important life stressor leading to increased symptomatology in kids. But there are several other plausible explanations. Yet another explanation not described here is the possibility that custodial parents (usually mothers) are more stressed after divorce and are less able to tolerate certain behaviors in their child:

Ritalin use doubles after divorce, study finds

By Scott Anderson Tue Jun 5, 9:47 AM ET

TORONTO (Reuters) - Children from broken marriages are twice as likely to be prescribed attention-deficit drugs as children whose parents stay together, a Canadian researcher said on Monday, and she said the reasons should be investigated.

More than 6 percent of 633 children from divorced families were prescribed Ritalin, compared with 3.3 percent of children whose parents stayed together, University of Alberta professor Lisa Strohschein reported in the Canadian Medical Association Journal.

The study of more than 4,700 children started in 1994, while all the families were intact, Strohschein said. They followed the children’s progress to see what happened to their families and to see what drugs were prescribed.

“It shows clearly that divorce is a risk factor for kids to be prescribed Ritalin,” Strohschein said.

Other studies have shown that children of single parents are more likely to get prescribed drugs such as Ritalin. But is the problem caused by being born to a never-married mother, or some other factor?

“So the question was, ‘is it possible that divorce acts a stressful life event that creates adjustment problems for children, which might increase acting out behavior, leading to a prescription for Ritalin?”‘ Strohschein said in a statement.

“On the other hand, there is also the very public perception that divorce is always bad for kids and so when children of divorce come to the attention of the health-care system — possibly because parents anticipate their child must be going through adjustment problems — doctors may be more likely to diagnose a problem and prescribe Ritalin.”

Ritalin, known generically as methylphenidate, is a psychostimulant drug most commonly prescribed for the treatment of attention-deficit hyperactivity disorder in children.

There is a big debate in much of the developed world over whether it may be over-prescribed — given to children who do not really need it. In March, a University of California, Berkeley study found that the use of drugs to treat ADHD has more than tripled worldwide since 1993.

Strohschein said it is possible that some mental health problems pre-date the divorce, so “it is possible that these kids had these problems before, but are only being identified afterward.”

Her study was not designed to find out why the children were prescribed the drug.

“I might be finished with the survey, but I am not necessarily finished with the question,” she said in a telephone interview.

Add comment June 6th, 2007

Lancet Iraq mortality study author Gilbert Burnham interviewed by New Scientist

The New Scientist magazine has an interview with Lancet Iraq mortality study author Gilbert Burnham:

>

Winning the war for Iraq’s dead

Counting the dead in war zones is what epidemiologist Gilbert Burnham and his team do for a living. But last year when they said 600,000-plus Iraqis had been killed in the war, the US, UK and Iraqi governments furiously attacked their figures for being far too high - though it turns out that UK experts agree with Burnham. Celeste Biever caught up with him recently

BY: Celeste Biever

There were already estimates of the dead in Iraq. Why did you decide to go ahead with your survey?

Our intentions were not political. Our centre is for refugee and disaster studies and this is simply the kind of thing we do. Other counts, such as the Iraq Body Count, which consists of volunteer academics and activists based in the UK and the US, rely on reports of deaths in the English-language press, but the press is in the business of producing news, not statistics. The IBC uses news reports mainly written in English, by people who can’t leave a very narrow area of Baghdad, while violence is worse in the Al Anbar and Diyala provinces. Mortuaries provide figures but a lot of bodies don’t make it there. Also press accounts and mortuary numbers record violent deaths, but people die in war from many causes.

Your figure was an order of magnitude higher than the IBC’s. Why should we trust your method?

Because it’s probably the best one for measuring the burden of conflict on the population. It’s used worldwide: in the Congo, in Banda Aceh after the Asian tsunami, for mortality data in Darfur, Angola and Uganda. And one of my former students, Paul Spiegel, used the technique to measure Serb activities against the Albanians in Kosovo. It was used as evidence in the trial of Slobodan Milosevic.

Why do you think your survey has been criticised?

These are unpleasant results, and they are associated with a war that has seriously divided the countries participating. Some people felt that we were not supporting the troops and were unpatriotic. I am not angry about that. As malicious as some of the hate mail I received is, I can see their point of view because I was in the military, in a combat unit in Korea during Vietnam. These soldiers in Iraq are volunteers, by and large, with good intentions, and they find themselves in a very difficult environment. As epidemiologists, we can produce the numbers, a good explanation for our methods and even a pretty strong statement on what they mean, but getting them accepted in policy circles and in people’s thinking takes time and is often difficult.

How did it feel to have the president attack you?

It’s not surprising to get criticism from people closely identified with the war. On the other hand, public health research often sends people to sleep so it was gratifying in an odd way to be associated with research that grabbed attention, especially heads of state.

You’ve said you will release the raw data to scientific groups who apply, “scrubbed” of the neighbourhoods where it was collected to avoid identifying the interviewees. Will this help?

I don’t know. Much of the criticism is based on unhappiness with the results. A repeat analysis won’t turn the figure from 600,000 to 60,000. Our intent is to be more transparent. We believe we will see numbers that are fairly consistent with ours. I received a lot of supportive emails from people who admired the courage of the team so I think many people already believe our figure.

What was your methodology?

We did a “cluster” survey, where we divided the country into clusters, picked a certain number from each province at random and sent Iraqi doctors to knock on doors in those clusters to ask how many people in each household (who had lived there at least three months) had died from any cause. We used that to produce a death rate for the clusters and then for the population of the whole country. The key is to try to be sure that you talk to enough people and you don’t have biases in selection.

How did Iraqis react to the interviewers?

The general sense the interviewers had was that people were happy to talk to them. They felt gratified that someone was asking and were eager to talk about their experiences.

Could you trust what people told interviewers?

People might forget exact dates, but death is a big event, so they don’t tend to forget that. To double-check, our interviewers asked for death certificates. Ninety per cent of the people who were asked produced them. Of course, a certificate doesn’t stop you hiding deaths: one could imagine households might be reluctant to mention it if someone got killed while involved in criminal activity or sectarian violence. But then the result would be an underestimate, not an overestimate.

Were there things you wanted to know you couldn’t get the interviewers to ask?

We were afraid, for example, to ask how people died as it might have made us look like we were representing a group looking for targets - and that could have endangered the interviewers. We also did not distinguish non-combatants from active combatants because asking that question was far too sensitive.

Were there other limits on your methodology?

Concern for the safety of our interviewers helped determine survey design. Coming up with a death estimate per governorate would have been the best but it would have required more clusters, and since each cluster has a risk associated with it we opted for a national figure. Also, we couldn’t use GPS devices as we had in 2004, where we randomly selected a GPS coordinate in each cluster and used that house as a start point. With more car bombs set off remotely, the team was concerned that if they were spotted holding a GPS receiver their life expectancy would be fairly short.

What did you do instead?

We went back to what we did before GPS. The interviewers wrote the principal streets in a cluster on pieces of paper and randomly selected one. They walked down that street, wrote down the surrounding residential streets and randomly picked one. Finally, they walked down the selected street, numbered the houses and used a random number table to pick one. That was our starting house, and the interviewers knocked on doors until they’d surveyed 40 households. It was more complicated than using GPS but not inferior: the results were very close to the GPS survey. The team took care to destroy the pieces of paper which could have identified households if interviewers were searched at checkpoints.

Why didn’t you accompany the interviewers?

I don’t speak Arabic and I don’t look like an Iraqi, so my chances of surviving very long were not strong. Our Iraqi colleagues said it would endanger them too. We met in Jordan to design the survey, at the end to begin analysis, and kept email and phone contact during.

Were the interviewers willing to risk their lives?

They knew there was a risk. Some dropped out before we started, but once we started, everybody stuck it out. There was a strong feeling of professionalism, and I take my hat off to them. These were the most courageous people in the whole operation. The rest of us took flak for the survey, but that’s nothing compared to their courage. We were very worried about someone dying. We took all the safeguards we could, and I tracked what was happening very closely throughout the three months it took. I remember the day word came back we had finished the last cluster and all eight interviewers, their supervisor and drivers were back safely. I was just elated.

Profile

Gilbert M. Burnham trained as a doctor, then went on to manage health services and oversee research in Zambia and Malawi for 15 years. He is now co-director of the Center for Refugee and Disaster Response at the Johns Hopkins Bloomberg School of Public Health in Baltimore, Maryland.

[h/t MediaLens Message Board. Original link requires subscription.]

12 comments April 23rd, 2007

Iraq Body Count trashes Lancet Iraq mortality study yet again

The Iraq Body Count (IBC) folks have written a letter to the World Socialist Web Site critiquing the site for having referred in a recent article to the 600,000+ excess Iraqi deaths estimated in the October 2006 Lancet Iraq mortality study. The IBC letter is rather silly, objecting, for example, to the fact that the mortality study was a “cluster survey”:

Secondly, the Lancet researchers visited 47 neighbourhoods and conducted interviews in 40 adjoining households in each neighbourhood. Only about 1,800 households containing 12,000 Iraqis were surveyed. These households reported a total of 302 violent deaths, each of which has been multiplied by two thousand to provide an estimate of how many of Iraq’s estimated 26 million population would have died if this proportion of deaths were representative of the country as a whole.

The IBC folks contrast the dead counted in the mortality survey with the dead reported in the media that they count:

The Lancet researchers documented only 300 violent deaths. Iraq has reached such a sorry state that IBC records 300 deaths every few days.

Correct. So what’s the point here? Is it better to precisely count the wrong thing (deaths reported in the media) than to use survey methodology to somewhat imprecisely count what one is interested in knowing? The IBC folks sure seem to think so.

IBC also repeats another faulty argument they have previously made:

The study’s central estimate of over 600,000 violent deaths seems exceptionally high. Even its lower bound 95 percent confidence interval of 426,000 violent deaths is shockingly high. It is very unlikely that incidents of this scale would be so consistently discounted by the various media in Iraq. Although IBC technically requires only two sources for every corroborated death in its database, we actually collect, archive and scrutinize every single report we can find about each incident before it is added to our database. For larger incidents, the number of reports can run into the dozens, including news published in English, in the original, and others, mostly the Iraqi press, published in translation. In IBC’s news archive for August 2006, the average-size attack leaving 5 civilians killed has a median number of 6 reports on it.

Their argument on the number of press reports for large incidents would only be valid if the press covered the whole country. Yet, there is a disproportionate proportion of the press around Baghdad and, to a certain extent, a few other cities. So, their claim here can also be understood as being due to this press running to the same few incidents to which they have access.

Further, the IBC claim here ignores the Lancet mortality study finding that the largest cause of deaths, indeed 56% of them, were due to gunshot wounds, which, by their nature, are not necessarily grouped in large incidents with multiple press reports.

The IBC folks make another rather odd point:

We would hope that, before accepting such extreme figures, serious consideration is given to the possibility that the population estimates derived from the Lancet study may be flawed. The most likely source of such a flaw is some bias in the sampling methodology such that violent deaths were vastly over-represented in the sample. The precise potential nature of such bias is not clear at this point. But to dismiss the possibility of such bias out of hand is surely both hasty and irresponsible.

I agree that “serious consideration is given to the possibility that the population estimates derived from the Lancet study may be flawed.” I further agree that studies with “extreme figures” should be examined closely. I also agree that sampling bias is the most likely place to find potential bias. And of course, one should not “dismiss the possibility of such bias out of hand.” But I further believe that seeking to dismiss a major study by pointing to”some bias in the sampling methodology” when “the precise potential nature of such bias is not clear at this point” is indeed “surely both hasty and irresponsible.”

The main sampling bias that has been raised so far is the so-called main street bias (MSB), having to do with alleged oversampling of streets intersecting main streets. It is possible that there may be some degree of such sampling bias, though the mortality study authors strenuously deny this. . But the MSB authors derive a bias factor of three by making extreme, unsupported, assumptions regarding their mathematical model’s basic parameters. Even if one were to make these extreme assumptions, a bias of three would still leave excess Iraqi deaths at over 200,000, way above the 60,000 that these authors from IBC appear to be defending. Any reasonable estimate of these parameters would lead to a much smaller bias factor and an estimate of excess deaths considerably higher.

It is possible, of course, that there are other, so far undiscovered, sampling biases in the Lancet study. As every scientist knows, any study may have unknown biases. But the possibility of unknown biases is not a plausible nor honorable reason to dismiss an apparently well-conducted study.

While encouraging skepticism about the Lancet mortality study figures, these IBC authors neglect to mention the virtual certainty that the IBC numbers are too small. After all, it strains credulity that, in circumstances of war, that all deaths would be reported in the media. In fact, IBC founder John Sloboda said in an April 2006 BBC interview:

We’ve always said our work is an undercount, you can’t possibly expect that a media-based analysis will get all the deaths. Our best estimate is that we’ve got about half the deaths that are out there.

I guess the fact that IBC counts are underestimates doesn’t fit the “message” IBC so carefully strives to create and so can safely be neglected when criticizing others. As far as can be told, IBC makes almost no effort to correct mainstream media’s routinely missing the fact of IBC’s numbers being undercounts of the true number of deaths. IBC routinely pleads lack of resources to correct such misinterpretations. It is to be wished that IBC would spent even a tiny fraction of the energy they’ve devoted to attacking the Lancet mortality study (see, e.g., here, here, and here), its authors and supporters, to correcting media misrepresentations of the meaning of IBC figures.

Skepticism, debate, and careful examination of potential biases is the heart of science. I maintain somewhat skeptical about the results of the Lancet mortality study. The results are so extreme that they deserve to be carefully examined. But I see no grounds to dismiss the study; the possibility of possible unknown bias is far from a reasonable basis for such dismissal. Amazingly, given the horrific conditions, other groups still are routinely conducting surveys in Iraq. It strains understanding why none of them has used the opportunity to provide an independent assessment of excess mortality. For replication is the only way that scientific debates are ultimately resolved. In the meantime, given all the evidence available at this point, it seems very likely that Iraqi excess deaths are in the hundreds of thousands, perhaps the high hundreds of thousands. Far too many have died as a result of the invasion and occupation and far too many will continue dying as the occupation and resultant resistance and civil war grind on.

NOTE: Tim Lambert at Deltoid has too additional items of interest. First, Les Roberts and Gilbert Burnham respond to a critical article in Nature. Second, there is an announcement that Lancet mortality study author Riyadh Lafta will speak this month at Simon Fraser on Friday, April 20, at 7 pm. Evidently, the United States denied Dr. Lafta a visa. I sure hope that his talk will be made available on the web.

12 comments April 7th, 2007

Next Posts Previous Posts


Pages

Calendar

November 2008
M T W T F S S
« Oct    
 12
3456789
10111213141516
17181920212223
24252627282930

Posts by Month

Posts by Category