Posts filed under 'Research Methods'

American Psychological Association tries to make money off of open access research depository, backs down

To many academics and researchers the American Psychological Association is largely known as the publisher of many high quality, and high status, journals. the National Institutes of Health recently required that all research funded y the NIH deposit publications in an open source depository. APA evidently tried to make money off this process, as the Chronicle of Higher Education reported yesterday:

July 15, 2008
Psychological Association Will Charge Authors for Open-Access Archiving

By Lila Guterman

In what appears to be a new policy, the American Psychological Association will require authors who publish in its journals to let it deposit their papers in open-access repositories — and it will charge them $2,500 to do so.

Researchers who have grants from the National Institutes of Health must deposit their published articles in the institutes’ online archive, PubMed Central. Last week the journal Nature and many of its offshoots announced that they would deposit their authors’ articles for them. Free.

Now the psychological association says that its authors “should NOT deposit” their own manuscripts, and instead should allow the group to do so. “The deposit fee of $2,500 per manuscript for 2008 will be billed to the author’s university,” the policy says.

Because the NIH does not charge a fee, that money is apparently going to the psychological association.

Open-access advocates like Peter Suber, a research professor of philosophy at Earlham College, expressed outrage. “It’s as bad as it looks,” he told The Chronicle. “This is not a good use of anybody’s money.” Depositing an article in PubMed Central, he said, is a “clerical job that can be done by a machine.”

The psychological association did not immediately respond to a request for comment from The Chronicle.

This report, or other negative reaction appears to hve led the APA to back down. When one follows the link to their web site, one now sees:

Document Deposit Policy and Procedures for APA Journals

A new document deposit policy of the American Psychological Association (APA) requiring a publication fee to deposit manuscripts in PubMed Central based on research funded by the National Institutes of Health (NIH) is currently being re-examined and will not be implemented at this time. This policy had recently been announced on APA’s Web site. APA will soon be releasing more detailed information about the complex issues involved in the implementation of the new NIH Public Access Policy.

APA will continue to deposit NIH-funded manuscripts on behalf of authors in compliance with the NIH Public Access Policy.

To continue with these charges could lead many more academics and researchers to leave the APA, as any such fees would be taken, directly or indirectly, by their universities out of the grants, reducing already limited research funds. And non-APA journals would instantly become more attractive publishing venues.

1 comment July 16th, 2008

Census Bureau to dissapear gay marriages

Most researchers recognize that research is inherently affected by social beliefs, norms, and practices. But the Census Bureau is about to give us an enormous demonstration. They have decided to remove all gay marriages from their marriage data for the 2010 census, despite laws in Massachusetts and California legalizing those marriges. Perhaps next they’ll remove “Islam” from the list of reportable religions.

U.S. Census Bureau won’t count same-sex marriages

By Mike Swift
Mercury News

Tens of thousands of same-sex couples are expected to marry legally in California by 2010, if a constitutional ban on gay marriage doesn’t pass at the polls in November.

But no matter what the voters decide, the official government count of the number of married same-sex couples in California is not in doubt. It will be zero.

The U.S. Census Bureau, reacting to the federal Defense of Marriage Act and other mandates, plans to edit the 2010 census responses of same-sex couples who marry legally in California, Massachusetts or any other state. They will be reported as “unmarried partners,” rather than married spouses, in census tabulations - a policy that will likely draw the ire of gay rights groups.

The Census Bureau followed the same procedure for the 2000 census, and it does not plan to change in 2010 even though courts in Massachusetts and now California have ruled gay men and lesbians can marry lawfully.

“This has been a question we’ve been looking at for quite a long time,” said Martin O’Connell, chief of the Census Bureau’s Fertility and Family Statistics Branch. “It’s not something the bureau could arbitrarily or casually decide to change on a whim, because our data is used by virtually every federal agency.”

The Census Bureau is not falsifying people’s responses, O’Connell said, because the bureau will retain people’s original census responses.

“We’re not destroying data; we are keeping that data,” O’Connell said. “We are just showing the data published in a way that is consistent with the way every other agency publishes their data.”The Census Bureau does not ask about sexual orientation, but it does ask people to describe their relationships to others in their household. If a respondent refers to a person of the same gender as their “husband/wife” on the 2010 census form, the Census Bureau will automatically assign them to the “unmarried partner” category. Legally married same-sex couples will be indistinguishable in census data from those who chose “unmarried partner” to describe their relationship.

Researcher’s view

Critics say the census plan will mask the records of legal, same-sex, married couples and therefore degrade the quality of the government’s demographic data.

“I just think it’s bad form for the census to change a legal response to an incorrect response,” said Gary Gates of the Williams Institute, a think tank at the University of California-Los Angeles law school that studies gay-related public policy issues. “That goes against everything the census stands for.”

Gates, a prominent demographer who was consulted by Census Bureau officials about counting legally married same-sex couples, said one result is that the census will undercount marriages in states with gay marriage. And because the bureau defines a “family” as two or more people related by birth, adoption or marriage, it also will remove many same-sex married couples from being counted as families.

“It’s a systematic hiding not only of married gay couples, but gay couples as families, which I would argue is a fundamentally political decision,” Gates said.

One recently married couple called the policy “frustrating.”

“It’s just another layer of the hurdles we have to jump, as far as our relationship being recognized,” said Jim Winstead of Hollister, who recently married his partner, Rodney Naccarato-Winstead. The couple have an 18-month-old son.

Gay rights groups, learning of the policy this week, were also critical.

“To have the federal government disappear your marriage I’m sure will be painful and upsetting,” said Shannon Minter, legal director for the National Center for Lesbian Rights. “It really is something out of Orwell. It’s shameful.”

A spokeswoman for ProtectMarriage.com, campaigning in favor of the constitutional ban, declined to discuss the census issue in detail, but said it illuminates how the legalization of gay marriage potentially could dictate policy changes on government.

“One of our campaign cornerstones will be the fact that if the initiative doesn’t pass that public schools will be forced to teach the difference between gay marriage and traditional marriage,” said Jennifer Kerns.

Bureau’s reasoning

A census technical note that explains the bureau’s rationale on counting same-sex partners for the 2000 census notes that the 1996 Defense of Marriage Act “instructs all federal agencies only to recognize opposite-sex marriages for the purposes of enacting any agency programs.”

O’Connell said the Census Bureau has been unable to find any federal agency that collects data on same-sex married couples. Changing the policy before the 2010 census also would be a huge and difficult logistical issue.

“The last thing anyone wants is to use the 2010 census as a trial run,” O’Connell said.

Gates said, however, that the limitations on access to people’s original responses will make it very difficult for private researchers to analyze raw data and back out the number of same-sex spouses in California or other states.

“It’s an official closet,” Gates said, “that the government has built.”


1 comment July 12th, 2008

Deaths in wars often greater than prior report: New study

As regular readers know, I have followed, and even been involved in, debates over the number of dead in the Iraq war and occupation. Estimates have ranged from the Iraq Body Count’s around 90,000 “documented civilian deaths” to the ORB survey’s estimate of over one million as of last August. While it is clear that the IBC estimate, based as it is on media reports, is an undercount, and likely a radical undercount, the magnitude of the true figure has remained a subject of controversy.

Reuters reports on a new study of prior wars which finds that mortality counts are almost always underestimates. While the new study did not examine Iraq, it lend support to the position that the true figures are in the several hundreds of thousands, and could easily be higher.

Parenthetically, the new study estimates that 3.8 million people died in the Vietnam war, much higher than previous estimates, reminding us yet again of the horrors of imperial wars.

Here is the Reuters article. [The full study can be accessed here.]:

Deaths in Vietnam, other wars undercounted - study

By Will Dunham

WASHINGTON (Reuters) - New estimates of war deaths in 13 nations including Vietnam, Ethiopia and Bangladesh show that previous counts vastly understated the lives lost to war in the past half century, researchers said Thursday.

The new estimates relied on data from nationally representative population surveys done by the U.N. World Health Organization in these countries earlier this decade to calculate death tolls in wars waged from 1955 to 2002.

In most of the countries, this method pointed to much higher loss of life than broadly cited media estimates of the various war death counts had shown, the researchers said.

For example, the method indicated 3.8 million Vietnamese died in the protracted fighting in Vietnam, mostly from 1955 to 1975, compared to previous estimates cited by the researchers of 2.1 million.

Christopher Murray of the University of Washington said the findings, published in the British Medical Journal, suggest standard ways of tracking war deaths using media, eyewitness and combatant accounts tend to underestimate deaths, particularly in smaller wars.

Murray, who heads the university’s Institute for Health Metrics and Evaluation, and colleagues designed a method of figuring violent war-related deaths using data on siblings of respondents in large household surveys conducted later in peacetime.

Random samples of people in the 13 countries were asked about their brothers and sisters, including whether they had died of wartime injuries. The researchers then extrapolated the data to come up with national death toll estimates.

In Ethiopia, the method indicated there have been 579,000 wartime deaths, higher than the previous estimate cited in the study of 275,000. In Bangladesh, the toll was put at 269,000, up from the previous estimate of 58,000.

Country by country, on average, the old estimates were about three times lower than the new ones.

In the 13 countries combined, the new method figured there were 5.4 million deaths from 1955 to 2002, topping the previous combined estimates of 2.8 million, the researchers said.

In Bosnia, the researchers figured 176,000 war deaths, up from the 55,000 previous estimate. In Sri Lanka, the new estimate was 215,000 deaths, compared the previous estimate of 61,000.

In Zimbabwe, the new estimate was 141,000 war deaths, compared to the previous estimate they cited of 28,000.

Other countries examined in the study were Myanmar (Burma), Georgia, Guatemala, Laos, Namibia, the Philippines and the Democratic Republic of Congo.

The study did not look at war dead in Iraq or Afghanistan.

Ziad Obermeyer of Brigham and Women’s Hospital and Massachusetts General Hospital in Boston, another of the researchers, said accurate estimates of death tolls during wartime are extremely difficult to make.

He also said the findings undercut the idea that the advent of modern weapons like “smart bombs” had made war less lethal. (Editing by Todd Eastham)

Add comment June 20th, 2008

Journal of Research Practice looking for papers

The Journal of Research Practice is a relatively new transdisciplinary journal seeking to bring together researchers from diverse fields to explore issues of research practice. JRP is open access, with a truly diverse and international editorial group. [Note: I have been on the Editorial Advisory Board since its first issue.] JRP is actively seeking high quality submissions.

All submissions should be written for a wide audience and should explore new ideas or experiences regarding the practice of research. As a reviewer, I’ve noticed that a major reason papers are rejected is because they are written to assume intimate knowledge of a particular research domain and are not obviously relevant to researchers from other disciplines.

A Call of Submissions can be downloaded here. Please download and post it. And do seriously consider submitting papers.

Add comment May 7th, 2008

Statistical tools help guide responses to human rights crises

Science News discusses the complexities of using statistics to guide humanitarian responses, using the issue of estimating Iraq mortality as an example:

Humanitarian Statistics
Statistical tools help guide responses to human rights crises

Julie J. Rehmeyer

In late 2006, a statistical study of deaths that occurred after the invasion of Iraq ignited a storm of controversy. This Lancet study estimated that more than 650,000 additional Iraqis died during the invasion than would have at pre-invasion death rates, a vastly higher estimate than any previous. But in January, a World Health Organization study placed the number at about 150,000.

The conflicting findings highlight just how difficult it is to gather reliable information in a war zone. But they also show the increasing involvement of statisticians in informing responses to humanitarian crises. In addition to the work in Iraq, statisticians have gathered evidence that has aided in the prosecution of Slobodan Milosevic, guided reparations for the civil war in Sierra Leone, and helped determine the needs of Katrina survivors, among many other projects.

“You can go to a congressional hearing or an international war crimes tribunal and you can hear the stories,” says Lynn Lawry of the International Medical Corps. “But how many are we talking about? How many people are at risk? How many people are affected?”

Statisticians are well-suited to answer these questions because they have the tools to put together partial information into a global picture. For example, even if complete records can’t be gathered, a statistician can survey a small number of randomly chosen people affected by a crisis and infer from their experiences the likely impact on the population as a whole. For example, Jana Asher of Carnegie Mellon University in Pittsburgh, Pa., developed an estimate of the rates of rape across Sierra Leone by determining how many women from a national sample had been raped.

But humanitarian crises pose huge challenges. Little information may be available—even from before a crisis—about how many people live where. Even if a previous census was taken, the high birth and death rates in developing countries tend to quickly make censuses outdated. Areas within continuing war zones can be unsafe for survey workers.

“When you have a displaced population that has been forced to flee their homes, all the traditional census methods really break down very badly,” says David Banks, a statistician at Duke University in Durham, N.C. “The refugees don’t have addresses. They’re wandering from one camp to another. Communication is poor.”

These challenges have to be met with very carefully designed protocols. For example, the Lancet study of Iraq, with the shockingly high mortality rates, was initially criticized for not surveying people who lived in back alleys because the areas were too dangerous for surveyors. Les Roberts, who was at Johns Hopkins University in Baltimore at the time but is now at Columbia University, and his collaborators on the study argued that the critics had misunderstood their randomization technique.

Random surveys are not the only useful statistical method. To tally the number of deaths related to the conflict in Timor-Leste, Romesh Silva and Patrick Ball of the Human Rights Data Analysis Group combined incomplete datasets to generate a broader picture of events. The Indonesian military claimed that its occupation of Timor-Leste had caused no deaths. Many stories had been told of killings and famine, but Silva and Ball wanted solid evidence.

Along with gathering about 8,000 personal accounts conveyed to the Commission for Reception, Truth and Reconciliation, Silva and Ball conducted a census of public graveyards including 319,000 gravestones and a survey of a random sample of 1,400 households about displacements and deaths. The researchers found that the different lines of evidence corroborated one another strongly, adding to the strength of each approach. In addition, Silva and Ball could observe how often names recurred across the different databases and get a much better estimate of the total number of deaths across the country.

They found that Indonesian occupation of Timor-Leste from 1974 to 1999 led to more than 100,000 deaths beyond what would have been expected in peacetime, through a combination of direct killings, famine, and illness.

The conflicting studies in Iraq show just how tricky it is to apply these methods in messy real-life situations. About the Lancet study, Asher says, “I don’t think there was anything obvious in what they did that someone can point to and say this method is flawed. But the WHO study used appropriate methodology too.”

The most suspect part of the Lancet study, Asher says, is that the researchers didn’t supervise the survey workers closely. On the other hand, the World Health Organization relied on government workers to administer the questionnaires. People can be intimidated by government workers and be less inclined to say much, a phenomenon that is particularly common in unstable countries. The only way to resolve the conflict, Asher says, is to do yet another study, with an even more careful design.

If you would like to comment on this article, please see the blog version.

References:

Asher, J., D. Banks, and F.J. Scheuren, eds. 2008. Statistical Methods for Human Rights. New York: Springer. See www.springer.com/statistics/social/book/
978-0-387-72836-0.

Iraq Family Health Survey Study Group. 2008. Violence-related mortality in Iraq from 2002 to 2006. New England Journal of Medicine 358(Jan. 31):484-493. Available at http://content.nejm.org/cgi/content/full/358/5/484.

Burnham, G. . . . and L. Roberts. 2006. Mortality after the 2003 invasion of Iraq: A cross-sectional cluster sample survey. Lancet 368(Oct. 21):1421-1428. Abstract available at http://dx.doi.org/10.1016/S0140-6736(06)69491-9.

Silva, R., and P. Ball. 2006. The Profile of Human Rights Violations in Timor-Leste, 1974–1999. A report by the Benetech Human Rights Data Analysis Group to the Commission on Reception, Truth and Reconciliation of Timor-Leste. Available at www.hrdag.org/resources/timor_chapter_graphs/
timor_chapter_page_01.shtml.

Add comment March 31st, 2008

NPR downplays Iraqi dead

FAIR has issued an Action Alert: NPR Underreports Iraq Deaths, dealing with an NPR report by Scott Simon in which he stated:

“This coming Wednesday marks the fifth anniversary of the start of the war in Iraq. So far 3,975 U.S. service men and women have died. Estimates on the number of Iraqis killed range from 47,000 to 151,000, depending on the source.”

These numbers are, of course, silly. The 151,000 presumably comes from the recent World Health Organization/Iraqi Ministry of Health study recently reported in NEJM. FAIR speculates that th 47,000 is from Iraq Body Count, but it is their estimate of those killed as of June 2006 [In the email below I erred and said August] and is considerably higher now, around 85,000. And other studies from the Lancet and the British polling firm ORB yield far higher estimates of around one million [extrapolating the Lancet study]. Thus, the number of dead from violence is almost certainly at least 250,000 and most likely higher, perhaps far higher. NPR miserably failed its listeners, the Iraqi people, and the truth in this instance. Alas, this is far from the only time that NPR has been a vessel for propaganda supporting the war.

FAIR calls upon concerned listeners to write the NPR ombudsman and ask for an investigation. Here is my email:

I hope that you will look into the very misleading figures in the March 15 braodcast in which Scott Simon described estimates of Iraqis killed since the war began as from 47,000 to 151,000. As a researcher, I have followed this area closely. I can imagine no credible source for the 47,000 figure as Iraq Body Count (IBC, which counts those dead reported in the Western media, puts the current figure of such reported deaths as over 80,000.  IBC is certainly a radical undercount given the exigencies of reporting in a war-torn country where over 100 reporters have been killed and many others kidnapped or arrested.

Further, the 151,000 figure, from the World Health Organization and Iraqi Ministry of Health, was as of August 2006, before the most intense violence.

Further, several additional studies from Johns Hopkins epidemiologists (published in the Lancet) and from the British ORB polling organization have arrived at far higher figures. Johns Hopkins estimated around 600,000 victims of violence by summer 2006 and the ORB estimated around 1,000,000 by the end of 2007.

Surely NPR listeners, as they weigh the five years of war deserve accurate information on the current state of knowledge on the true costs of that war.This Ameriacan Life has reported on the Lancet studies. Surely over reporters should as well. Much as I love Scott Simon, in this case, his report was grossly deceptive at best. The purpose of NPR is to create an informed citizenry. In this instance you failed your mission.

Please investigate and make sure that such an egregious error does not recur.

Thank you very much.

Post your email here.

Add comment March 26th, 2008

Open access now mandated at Harvard

Revere at Effect Measure reports that Harvard is now mandating that all faculty research in the School of Arts & Sciences be made available to the public immediately.

“The Harvard requirement mandates immediate free publication online in a Harvard hosted repository, searchable by Google and other search engines. Thus Harvard authors are not supposed to publish from now on in some extremely high profile journals like Nature and Science who prohibit fee access of papers for a period of time after publication. Whether these journals will publish Harvard papers under these conditions now is a question we don’t know the answer to. It could get very, very interesting.”

This is good news for Open Access, which I support in principle. But there  is a problem. Many of us do much of our research and writing without funding. Open Access generally works by having the authors pay. When there is a funded grant, this is fine. But for some of us in smaller institutions which won’t pay te cost, this is a disincentive. I imagine that scholars in third world countries will also have problems with cost. Some mechanism needs to be developed to distinguish between funded and non-funded research.

2 comments February 16th, 2008

ORB polling firm reissues Iraq mortality estimate of one million dead

Last September, the British polling firm ORB issued a report estimating that 1.2 million Iraqis had died. After criticism, ORB announced that they would conduct additional surveys in rural areas to check their results. the implication was that they had undersampled rural areas, which might have inflated their mortality estimate. At that time, they stated that they expected the additional results to be available by early October. Well it’s now late January and they have just released their revised results. They now estimate that their estimate of 1.2 million deaths should be revised downward to 1,033,000 with a range of 946,000 and 1,120,000. Here is their press release:

New analysis ‘confirms’ 1 million+ Iraq casualties

January 28th 2008

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Further survey work undertaken by ORB, in association with its research partner IIACSS, confirms our earlier estimate that over 1,000,000 Iraqi citizens have died as a result of the conflict which started in 2003.

Following responses to ORB’s earlier work, which was based on survey work undertaken in primarily urban locations, we have conducted almost 600 additional interviews in rural communities. By and large the results are in line with the ‘urban results’ and we now estimate that the death toll between March 2003 and August 2007 is likely to have been of the order of 1,033,000. If one takes into account the margin of error associated with survey data of this nature then the estimated range is between 946,000 and 1,120,000.

Our revised estimate – which compares to a figure of 1.2 million published in August 2007 – is based on a representative sample of 2,414 adults aged 18+. They were asked the following question:-

How many members of your household, if any, have died as a result of the conflict in Iraq since 2003 (i.e. as a result of violence rather than a natural death such as old age)? Please note that I mean those who were actually living under your roof?
None

72%

One

14%

Two

3%

Three

1%

Four or more

*

Don’t know

2%

No answer

8%

  • * = figure more than zero but less than 0.5%
  • Note: Of the 251 people who declined to give an answer the large majority (66%) were interviewed in Baghdad.

Casualties Calculation:

Among the over 2,160 respondents who answered the question 20% said that there had been at least one death in their household as a result of the conflict which started in 2003. Within these households the average number of deaths was 1.26 people.

The last complete census in Iraq conducted in 1997 indicated a total of 4,050,597 households. Based on this our data suggests a total of 1,033,239 deaths since March 2003. Given that the statistical margin of error on a sample of approximately 2,160 people is +1.7% (for findings at or near 20%) the possible range of casualties implied by our data is:

 

Estimated number of deaths from conflict since 2003

Margin of error on finding at or near 20% (on sample of c. 2,160)

Maximum estimated number of deaths from conflict since 2003

Minimum estimated number of deaths from conflict since 2003

% households with deaths

20.2%

1.7%

21.9%

18.5%

Mean casualties per household (of those with at least one death)

1.26

 

1.26

1.26

Est. # h/holds in Iraq

4,050,597

 

4,050,597

4,050,597

Estimated total number of casualties

1,033,239

 

1,120,220

946,258

Base: 2,163 Iraqi adults answering question

Detailed analysis (which is available on our website) indicates that over two-fifths of households in Baghdad have lost a family member, higher than in any other area of the country. Meanwhile among those willing to declare their doctrine (and for quite obvious reasons about half those interviewed prefer to simply describe themselves as Muslims) those from Sunni households (33%) were significantly more likely to say the conflict had claimed a household member. The respective figure for Shias being half that figure (16%).

Research Methodology:

§ Results based on face-to-face interviews amongst a nationally representative sample of 2,414 adults aged 18+. Interviews conducted throughout Iraq - 1,824 in urban areas and 590 around rural sampling points.

§ The survey methodology utilized multi-stage random probability sampling and covers fifteen of Iraq’s eighteen governorates. Overall 112 unique sampling points were covered – 92 in urban areas and 20 in rural locations.

§ For reasons surrounding interviewer safety Karbala and Al Anbar were not included in this research. Irbil is also excluded as the local authorities refused our fieldwork team a permit to operate. We feel that the net result of these exclusions –two areas of relatively high volatility since 2003 and one relatively stable - is that the casualty estimates reported are unlikely to overstate the actual figure.

§ The first batch of interviews was completed August 12th – 19th 2007 with the ‘rural booster’ conducted 20th – 24th September, 2007.

§ At the 95% confidence level the ‘margin of error’ on the sample who answered (2,163) is +2.1%. This figure is applicable to findings at or near 50% while for findings in the region of 20% this margin drops to +1.7%

§ Full results and data tabulations are available at www.opinion.co.uk/newsroom

§ IIACSS (Independent Institute for Administration and Civil Society Studies) is a polling/ research company established in Iraq in 2003 and which has a network of interviewers covering all regions of the country. Further information about IIACSS and its founding director Dr. Munqith Dagher can be found within the relevant news article in the Newsroom section of ORB’s website.

§ ORB is a member of the British Polling Council and abides by its rules.

For further information please contact Allan Hyde on 020 7611 5270 or email ahyde@opinion.co.uk

The Opinion Research Business

34 Bedford Row

London

WC1R 4JH

Tel: 020 7611 5270

www.opinion.co.uk

See also their:

New Casualty Tabs.pdf
MRS story.pdf

Add comment January 28th, 2008

Tirman: Right-Wingers Can’t Cover Up Iraq’s Death Toll Catastrophe

John Tirman describes the right wing hatchet job on the Lancet Iraq mortality study in detail in Alternet:

Right-Wingers Can’t Cover Up Iraq’s Death Toll Catastrophe

by John Tirman

Now I know what Hillary Clinton meant, first hand, by that “vast right-wing conspiracy.” When the Wall Street Journal editorial page and the Sunday Times in London are going after you — along with about 100 right-wing bloggers — rest assured you’ve hit a nerve.

Or is it just Soros Derangement Syndrome at work?

More than two years ago, I commissioned a household survey of Iraq to learn how many people had died in the war. This topic had been virtually ignored by the news media and the U.S. government. It was important to know for at least three reasons. The first was to try to understand the nature of the violence there, which was steadily growing and creating a humanitarian crisis, possibly a regional conflagration. Second, it might tell us something about how and when to exit. Third, we needed to know for the sake of our national soul. What had we wrought?

So I contacted the people who had done a previous, largely ignored survey-top public health professionals at Johns Hopkins University. They had published a survey in October 2004 that showed 98,000 had died in the first 18 months of the war, which was greeted with disbelief and charges of politicizing science, and quickly dismissed.

I said: ‘do a bigger survey to improve the accuracy, and I will make sure it gets the proper attention in the news media.’ They did do a bigger survey, and I managed a public education campaign that permitted the results to be considered more broadly, results that estimated total deaths at 600,000 by violence after 40 months of war. The survey was published in The Lancet, the British medical journal. And get attention it did, roundly disbelieved and scorned by war supporters, but spurring a brief but intense debate about the human cost of the war.

Dozens of statisticians and other professionals scoured the study and its data to see if the methods and implementation were proper; a special committee at the World Health Organization was convened to review it, and the Lancet had also subjected it to rigorous peer review. The survey held up to this scrutiny, with quibbles and some lingering “should have done this” and “might have done that.” But virtually every competent person agreed that the study provided the best estimate we have.

Then, earlier this month, the National Journal, a Capitol Hill “insider” weekly, ran a cover story titled “Data Bomb” by Neil Munro and Carl Cannon. In a note by Munro published by the National Review blog, he asserts:

“George Soros funded the survey. The U.S. authors played no role in data-collection, and did not apply standard anti-fraud measures. The chief Iraqi data-collector had earlier produced medical articles to help Saddam’s anti-sanctions campaign in the 1990s, and said Allah guided the prior 2004 Lancet/Johns Hopkins death-survey. Some of the field surveyors were employed by Moqtada Sadr’s Ministry of Health. The Iraqis’ numbers contain evidence of fakery, and the Lancet did not check for fakery.”

It’s a neat summary of their allegations, which include dozens of unfounded charges, promiscuous innuendo, misquoting of the principals, and misunderstanding statistics, and relies on two disgruntled critics. It was a hatchet job, pure and simple. Not a sentence of Munro’s summary is truthful, and that goes for much of the NJ article, too, which I have demolished elsewhere (PDF). The principal author, Gilbert Burnham, M.D., Ph.D., and his colleagues have taken time from their clinics in Afghanistan and Jordan and Africa to answer the charges on the John Hopkins website, too ( with a letter here, and a FAQ here).

But lies have a way of proliferating on the Internet, and so it was with this set of schoolyard bully brickbats. What seemed most to get under the skin of the right-wing media was a small grant for public education funded by the Open Society Institute, a foundation created by George Soros.

The charges of fraud that NJ clumsily made but never came close to proving were of course a tonic to the war supporters who were shamed by the estimate of 600,000 fatalities. There is nothing as devastating to the increasingly discredited case for war as the specter of the U.S. invasion having caused, directly and indirectly, more deaths than were attributed to the bloody reign of Saddam Hussein.

But it was news that “Soros” was a donor, and the wingnuts went berserk. The line that Munro and Cannon took was that “Soros” was somehow behind the survey from the start, which was timed to affect the 2006 elections. It was not only fraud, they contend, but the perversion of science for political ends backed by the disgruntled, Bush-hating billionaire.

It’s classic right-wing defamation, and of course none of it is true. Munro and Cannon were painstakingly walked through the chronology and donors, but deliberately ignored it to fashion their paranoid fairy tale, and the Wall Street Journal et al lapped it up.

We commissioned the survey on October 25, 2005, hoping to get it done as quickly as it could be done professionally, and perhaps have the results out in the spring. Why wait? But Iraq quickly became too violent to permit teams of questioners go out to 1,000 randomly chosen households. So it was not until late spring that they did begin the door-to-door work-still very perilous-and completed the survey in early July. It took another two months to enter the data, have biostatisticians at Johns Hopkins analyze it, and write up the article. The Lancet then took weeks to peer review. It was released when ready. There was no political agenda; there didn’t need to be. The results spoke for themselves.

The Open Society Institute came late to the process, announcing to me that a grant had been made for public education on May 4, 2006. That is six and a half months after the survey process began. We had already paid for the survey out of internal funds. Less than half of the cash needs of the survey, the analysis, and the public education effort was paid for by OSI. (If the real cost of the effort were totaled-to include salaries of Burnham, myself, and many others who were not compensated directly-then the OSI contribution would have offset about 10 percent of the cost.) I doubt very much whether George Soros himself was ever aware of the grant. OSI is a very large, humanitarian foundation, and their $46,000 grant to MIT is small by their standards.

And, needless to say, OSI and “Soros” had no influence over the initiation, conduct, or findings of the survey. Neither Burnham and his colleagues nor the Lancet editors knew OSI was one of the donors. The contract was with MIT.

I carefully told this to Munro on the telephone, and Burnham’s colleague Les Roberts emailed the same information to Cannon last autumn. Munro had asked, among other hostile questions, whether any Muslims or Arabs were supporting the survey, a racism reflected in his remark about Allah above and a charge in the NJ piece that the survey teams lacked American oversight and were thereby suspect. But he was emotionally fixated on Soros, and asked about his role repeatedly. When I tried to offer corroborating evidence for the survey, he screamed at me that none of that mattered. I could see where this was going.

Of course, Munro himself has been a rabid supporter of the war from the start. In the tradition of former NJ editor Michael Kelly, who called opponents of the war traitors, Munro agitated for the “destruction of Iraq” as early as November 2001. He had elsewhere insisted that the peace in Northern Ireland was the result of the British Army’s iron fist. His sentiments were on display through the hatchet job on us, not least in alleging that The Lancet article was a spur to jihadists.

So the headlines-”Soros Underwrites Osama’s Talking Points,” and “$oros Iraq Death Claim was a Sham” are typical. The Soros Derangement Syndrome derives, I suspect, from his special status as a traitor to his class, as the right used to refer to FDR. Someone so intelligent, articulate, actively compassionate, and rich cannot be tolerated.

In an odd twist, a new mortality survey-approvingly mentioned by the NJ piece-appeared earlier this month in the New England Journal of Medicine. Conducted by the Iraqi Ministry of Health, it found 151,000 deaths by violence as of June 2006, about the same period as the Lancet article. Newspaper coverage duly noted that their estimate was only one-quarter that of the Lancet. But a little digging would have revealed much more: the total deaths attributable to the war, non-violent as well as violent, was about 400,000 for that period, now 19 months ago. If the same trends continued, that total today would be more than 600,000.

The deaths-by-violence in that latter survey remained the same from year-to-year, however, which is not plausible-all observers agree that violent deaths were rising sharply in 2005 and 2006. The discrepancy is found in how the survey was conducted: interviewers identified themselves as employees of the Ministry of Health, then under the control of Shiite cleric Moktada al Sadr. Those interviewed, therefore, would be wary of saying a brother or son or husband had been killed by violence, fearing retribution. And, indeed, there are non-violent categories in the survey that suggest just such equivocation: “Unintentional injuries” would equal about 40 percent of the death-by-violence toll, for example. Road accidents were ten times their pre-war totals-if someone is run off a highway by a U.S. convoy, is that a “non-violent” death?

The researchers, to their credit, acknowledge that their estimate is likely too low due to several factors. They did not go into dangerous neighborhoods, which made up 11 percent of the sample, and could not accurately estimate the death toll in those, which would of course have been high. Still, the survey is revealing on the non-violent mortality, too: deaths by kidney failure, cancer, diabetes, and others rose by several times, signaling the near-collapse of the health care system.

The MoH survey is the fifth trying to measure mortality during the war, and there is significant congruence among all. (The Lancet estimate is not actually the highest; that belongs to the private British polling firm, Opinion Research Business, which found that as of August 2007, 1.2 million Iraqis were dead due to the war.) But all the surveys point to one thing: a colossal amount of killing and dying has been going on, far more than numbers used in most discussions of the issue in the fleeting instances when concern for Iraqis appears.

And that, of course, should be the real issue here, not whether George Soros is interested in the issue. The NJ calumny and the many gleeful references to it are a sign that the pro-war legions are really at wit’s end. The catastrophe they created and supported must be blamed on others-the conveyors of bad news, the quisling liberals, and the Iraqis themselves.

But the dead in Iraq cannot be silenced as long as we have courageous researchers who will go into the warzone to gather data and tell us the truth. That’s what five surveys-against perilous of odds-have done, and the findings should haunt us every day.

John Tirman is Executive Director of MIT’s Center for International Studies.

1 comment January 21st, 2008

Pro Bono Statistics: 5 problems with the science of the IFHS study

The Pro Bono Statistics blog has some excellent pieces on the new NEJM Iraq Family Health Survey estimate of violent mortality. PBS raises several issues. First (s)he finds a correlation of .94 between governorate (province)) population size and sample size, which apparently contradicts the published description of the sampling procedure.

PBS also finds fault with the way the IFHS dealt with missing data, by extrapolating from Iraq Body Count data for two governorates.

While the more detailed postings described above are important to read, I’ll reproduce here a summary of five issues with IFHS raised by PBS. Evidently, PB is in the process of writing separate postings on each issue:

5 problems with the science of the IFHS study

Reviewing the IFHS study, I found 5 problems with the science of the study. I believe that taken together (but particularly the first three points, regarding the crucial role extrapolation plays in arriving at the estimates in the study, and regarding the ratio of under-reporting) those problems should be seen as grave. At the very least, they should be seen as putting the findings of the IFHS on equal or inferior footing to those of Burnham et al., rather than as being on superior footing due to the nominal large size of the sample in the IFHS.

I now give a brief abstract of the five problems. As I write a fuller description of each, I will add a link to it from the list here. Unless explicitly stated otherwise, death rates and counts below refer to violent deaths as defined by the IFHS authors.

1. Missing clusters and extrapolation using IBC numbers. The IFHS surveyors did not visit all of the clusters in their sample. Those areas that were judged to be dangerous went unsurveyed. A minority of those gaps (in Nineveh and Wasit) seem to be ignored, introducing potential bias. To fill in the rest of the gaps, the IFHS authors extrapolated from other areas. The extrapolation method was to calculate the mortality rate in all of Baghdad as a fixed factor times the mortality rate in some reference area, where the fixed factor was calculated using Iraq Body Count data. The same method (with a different factor) was applied to all of Anbar as well.

It is important to note that these extrapolations determined the total number of deaths estimated for Baghdad and Anbar. Any data that was collected within those areas was in effect ignored in calculating the death estimates. Thus the death count in Baghdad and Anbar, that together account for over 60% of the deaths in the estimate for the total, are purely a matter of extrapolation, and depend directly on the IBC extrapolation factors. To illustrate: the extrapolation factor used for Baghdad was 3.08. If instead the number was 6, that would have added about 80,000 deaths to the estimate.

The reliability of the IFHS estimate thus depends directly and substantially on certain properties holding for the IBC data (namely coverage rates which are constant across space and across political characteristics). We have no reason to assume that those properties hold, and have some reason to assume they don’t. The IFHS authors have apparently made no attempt to account for those issues - not so much even as to factor uncertainties into the size of the confidence interval.

In addition, the extrapolation method is the reason for the close resemblance, emphasized by the IFHS authors, between the IBC and IFHS breakdown of deaths by area. This resemblance is an artifact rather than a feature of the raw data and should not be seen as showing coherence between IFHS and IBC.

2. The extrapolation procedure is problematic even if the IBC extrapolation factors are assumed accurate. The extrapolation basis is the death rate in 3 reference governorates (the paper does not say exactly which, describing them only as the “three provinces that contributed more than 4% each to the total number of deaths reported for the period from March 2003 through June 2006″). Most governorates were sampled with 3 x 18 = 54 clusters each. Nineveh was sample with 72 clusters. Thus the estimate of deaths for Baghdad and Anbar (which, again, account for over 60% of the total) relies on at most 2 * 54 + 72 = 180 clusters. This number, much smaller than the nominal size of 971, is the dominant factor in determining the uncertainty of the estimate of the total (again, even if the extrapolation factor is assumed to be correct and known precisely).

This is the reason why the length of the confidence interval for the IFHS study (about 120,000 deaths) is not much smaller than that of Burnham et al. (about 370,000) despite the fact that Burnham et al. used only 47 clusters.

3. The IFHS does not account properly for uncertainty in under-reporting. In the same way that the IFHS estimate depends on the extrapolation factor, it depends on the assumed under-reporting factor. The justification for the factor used seems slim (I have not made an attempt to follow the reference given). Even accepting their assumptions - i.e., treating the proportion being reported as a normal variable with mean 0.65 and standard deviation of about 0.075, the authors fail to properly account for the uncertainty in the under-reporting in their calculation of the confidence interval of the estimate of the death rate. A proper accounting would increase the size of the confidence interval by about 25%.

4. In the IFHS paper, the heading “violent deaths” does not include certain types of injuries. I could not find this mentioned in the paper itself, but table 3 in the supplementary material and a statement by WHO official indicate that car accidents and “unintentional injuries” are not included in the estimate. This may seem reasonable a-priori regarding car-accidents, and to a lesser extent regarding unintentional injuries. However, contrary to the statement, those two categories account for more than a third of the deaths by injury in the survey. Also, there has been a dramatic increase in both of those categories as compared to pre-war rates. Under those circumstances, it appears unjustified to exclude these categories from the estimate. Including them in the estimate would increase it by more than 50%.

5. The last point is more of an indication of trouble (either in the methodology of the survey or in the way it is described in the paper) than a specific problem with the estimation. According to the description of the sampling method, 10 households were surveyed in each cluster, and there were (with few exceptions) 3 x 18 = 54 clusters per governorate. In such a set-up there should be no correlation between the number of people surveyed in each governorate and the size of the population in the governorate. However, looking at table 2 in the supplementary material of the paper, there appears to be a strong correlation between those two figures. It seems that the only way such a correlation could show up is the unlikely situation in which the size of the population in the a governorate is strongly correlated with the average household size in the the governorate.

Add comment January 20th, 2008

Previous Posts


Pages

Calendar

October 2008
M T W T F S S
« Sep    
 12345
6789101112
13141516171819
20212223242526
2728293031  

Posts by Month

Posts by Category