| For now, a controversy that has been bubbling over a crucial study around the subject has spilt over into the blogosphere, with the publication yesterday of this CIDRAP commentary Little evidence for New York City quarantine in 1918 pandemic by Mike Osterholm and John Barry, and today on EM with Non-pharmaceutical interventions for a pandemic: getting it right (with Addendum from John Barry).
First of all, I want to be honest right up front, and say that I have been an ardent supporter of the CMG measures, for reasons that I have written about before in various places but summarized here and here. But of course, as revere said, "The key point here is we shouldn't say something works just because we have nothing else to offer. If you are a public health scientist you should have a better reason for recommending something."
And if at any point, the evidence on which we base our public policy is in doubt, we must examine it with rigor, because, as Osterholm said so eloquently about his concern, This concern does not disprove that NPIs altered the course of the pandemic. But we in public health will face overwhelming challenges with risk communication and credibility during the next pandemic. While we will surely recommend the use of NPIs at that time, we have an obligation to society to tell exactly what we know and explain the science that supports our conclusions.
Now I'm no academic/researcher, and I'm certainly not qualified to pass judgment on the works and words of people so eminent as Howard Markel, Marty Cetron, and John Barry, who all rank high on the list of people I admire. What I'm doing here, is the same as what I've always done, which is attempt to 'translate' an issue into plain English for the general public, so that you can participate in examining the issues involved, and draw your own conclusions. Or perhaps the nature of the problem is such that no absolute conclusions can be drawn, only relative choices. I will leave that to the reader.
First the context:
The scientific case for the CMG measures (CMG is used here to designate the specific set of interventions recommended by the CDC, as opposed to a more general description of non-pharmaceutical interventions or NPI) is really based on 2 sets of data, one from computer models, the other from the study of NPIs used in the 1918 pandemic in US cities. Initially, only the modeling data were available. At the IOM meeting in October last year, and in the subsequent report, words of caution were raised that such interventions particularly school closure have serious consequences, and we mustn't base our policies on inadequate evidence.
Subsequent to the IOM report, 3 studies were published based on examination of historical records of public health measures used in 43 US cities and their possible correlation to outcome. I wrote a synopsis of the first two, from Hatchett, Mecher, & Lipsitch, and from Bootsma & Ferguson a little while back. The most recent one, the JAMA paper from Markel et al, I have only made some brief comments here (due as usual to time constraints). Suffice it to say that of the three, this is THE most authoritative study and thorough analysis of primary sources. The bibliography is published on the CDC site with 1,144 citations, many to primary documents. The important thing to note is that all 3 studies came to the same findings, that combinations of NPI that included school closures and banning of mass gatherings were correlated to reduction in weekly excess mortality, that timing of intervention was important for delaying the peak, reducing peak weekly excess mortality, and the total mortality, and that the effects were correlated to the duration of implementation.
Now for the controversy:
John Barry, eminent author of the book "The Great Influenza" wrote to the JAMA to say he believed that paper "has a serious problem in its data that calls its conclusions into question." Barry said he believed there was no evidence that New York city did impose a quarantine as Markel suggested, even though the New York city health commissioner Royal Copeland was quoted in the NY Times as saying they were putting people in quarantine, and that this puts into doubt the results and conclusions drawn from the rest of the study. He quoted 2 excerpts from the NY Medical Journal, one in which Copeland apparently argued against quarantine, and another in which his assistant cited various initiatives but these did not include imposing quarantine.
Markel et al wrote a response basically saying that Barry was wrong, that NY city started to quarantine suspected cases coming off ships even before local cases appeared, that the city made influenza a reportable and therefore actionable disease "in the same category of risk as cholera, bubonic plague, and smallpox", that the 'action' was quarantine as Copeland explained to the NY Times, and that 2 days later the NY Times reported 4 local people being quarantined. In addition, Markel said Barry had quoted the NY Medical Journal out of context, because Copeland in that article was talking about "the futility of potential home confinement of all New Yorkers and "universal" closure of public places." He also quoted an article from JAMA which described NY's quarantine actions. More importantly, Markel noted that Barry in his own book had written "New York was panicking, terrified. By now Copeland was enforcing strict quarantines on all cases."
Rather than giving you more blow-by-blow accounts, I think this gives you a flavor of the kind of issues being debated. The interesting development after that is the CIDRAP article, because it is one thing to write to the editor of JAMA, and quite another to take it to the internet. Writing to the editor is a common practice whenever published work is being queried, because in publishing a study, the editorial board does exercise the responsibility of carefully reviewing the study, including the methodology, before approving the manuscript for publication, and in a dispute would therefore act somewhat like a referee, both to ensure the fairness of the debate and ultimately protect the standard of the publication. It is not uncommon for such debates to go back and forth for a number of rounds. It is IMO a healthy way of getting at the truth, with the editorial board's actions or inaction passing ultimate judgment on the issue.
Be that as it may, the debate now moved to CIDRAP, which you can all read for yourself. Barry also wrote to revere at Effect Measure here. Barry expanded his queries about whether NYC did or did not impose quarantines, and said he himself had at first mistakenly believed that quarantine was used. The really interesting part for me is where he raised doubts on whether the city's commissioner of health was someone whose word could be trusted, based on the fact that he was a homeopath not an MD, and that he was a crony of a corrupt political elite. It's interesting because all of this is nothing new, it's all in Barry's own book, and yet he did initially believe in Copeland's words while writing the book, but has now apparently changed his mind.
In addition, he is also casting doubt on the findings from Chicago, saying that the overly large bibliography makes it impossible to determine where exactly the information on NPI in Chicago came from. He cited the "Chicago health department's 100-plus-page "Report of an Epidemic of Influenza in Chicago During the Fall of 1918, which is not in Markel's bibliography, only two actions were taken on day minus-2: The state banned public funerals and the city issued orders for teachers to inspect schoolchildren. These actions fall far short of the authors' metrics. The city's most tangible action actually did not occur until day plus-19 (21 days later)"
Now as I said, I make no claim to being able to determine which side has the 'right' answer, but I did make a little attempt in looking up some of this information. I'm in the process of trying to get my hands on the NY Medical Journal articles that Barry says he is happy to fax to anyone (alas, my fax is not working, so I'm asking a third party to help. I hope to get this information soon!)
Where I did have some success, is the straightforward approach of looking at the bibliography that is available for download here. To be honest I was a little intimidated by this 76 page 1,144 item list, as Barry said, but on further reflection, I took the simple approach of searching for the word 'Chicago'. What I was looking for was something that might look like primary data, to determine where Markel et al might have gotten their information. Out of the 22 results returned, I found these two listed:
J. D. Robertson, Report and handbook of the department of health of the city of Chicago for the years 1911 to 1918 inclusive (Chicago, 1919).
J. D. Robertson, Report of the department of health of the city of Chicago for the years 1919, 1920, and 1921 (Chicago, 1923).
What is interesting is that at least the first of these 2 documents appear to also be the source of data for the PNAS study by Hatchett et al. The supporting information for that paper can be found here. If you scroll down that page, you will find that for the city of Chicago, the PNAS authors had used the same document for the dates of intervention for the following:
- reportable disease
- isolation
- quarantine
- theaters
- Dance halls
- other closures
- private funerals
- protective sequestration
- ban on public gatherings.
Now this doesn't give us the dates in and of itself, but it does suggest to me that there IS an official source of information available giving detailed accounts of the type of intervention and the date of implementation, at least for the city of Chicago.
Where does this leave us? Frankly at this point I'm not too sure. Because the issue under debate appears on the surface to be about whether or not NY or Chicago implemented the measures as Markel suggested, or whether these cities had milder outbreaks because they had an earlier spring wave as Barry suggested. BTW, the CDC link does give us some really cool stuff, like this, which when you open it up is just a treasure-trove of information laid out perfectly for someone who is very visual like me. Because it is one thing to read this in the paper We found no statistically significant association of the EDR across the 43 cities when comparing successive waves. Specifically, the severity or occurrence of wave 1 is not associated, either positively or negatively, with the severity of wave 2; the severity of wave 2 is not associated with the severity of wave 3; and the severity of wave 3 is not associated with the severity of wave 4 and quite another to just see it on a chart, like this
In addition, if we were to go back to the big picture, and look especially at what Osterholm is saying, the issue is not just about the accuracy of the data, but also about whether we think there is enough justification for the measures recommended by the CDC. As I said, I'm a firm supporter, so count me out for an unbiased opinion, but what I'm interested in really is where Osterholm and Barry stand with regards to school closure. Because, the fact of the matter is, in all the debate about the CMG, none of the other measures such as voluntary home isolation and voluntary home quarantine and general social distancing ever came under dispute. The focus was always on whether or not we should close schools early enough (before 1% AR is achieved) or wait till children start falling sick or maybe dying.
Because, CIDRAP is one of the most credible sources for pandemic information, read by many public health officials up and down the country who will make decisions about whether and how far they want to implement the CMG recommendations, and Osterholm is a highly respected figure. His opinion counts. If for whatever reason he finds it hard to support early school closure, it may or may not be representative of other powerful opinions that we hear less often from, but I have a feeling that we may be in deeper water than many of us are comfortable with. If for that reason alone, I hope this controversy is resolved sooner rather than later!
UPDATE John Barry's views on NPI, cross-posted here.
video-link to interview |
