What’s the point of fancy statistics? Thinking about meta-analysis in medicine.

I recently finished reading Patient Care Under Uncertainty by econometrics giant Chuck Manski. He has turned his attention towards the surprisingly vexing issue of knowing if medical care actually works.  This is a favorite topic of mine, because a shocking among of medical care has little to no robust evidence that it works. Instead, it is based on unrealistic physiological models, unidentified observational studies and RCTs on completely non-representative samples of patients (at least, according to Vinay Prasad).

In his book, Manski offers numerous approaches to analyzing medical data that are conceptually more precise and sophisticated than the slightly hack-y statistics that plague modern medical research.

But most these approaches are dramatically more complex and difficult to implement. And I couldn’t help but wonder if all the work would be pointless. I strongly suspect that all of the conceptual nuance would be utterly lost in the translation to clinical practice. The new analyses would be more conceptually satisfying to econometricians and biostatisticians. But they would make more or less no impact on the actual practice of medicine, which is arguably the ultimate goal of all medical research.

To give a specific example: How should we combine data from multiple clinical trials to assess if medical therapies work and determine how well they work?

Manski’s Answer:

  • Don’t just take the average of the point estimates from different clinical trials (current standard). This hides the imprecision in the estimates and suppresses real uncertainty.
  • Reporting the range of point estimates is better, but unscientific. It reports variation in predictions across studies that have been reported in the literature. However, it doesn’t capture the potential variation that would be obtained using all plausible models, statistical specifications, data imputations, etc.
  • A better approach:
    • Get the data for each clinical trial (I should note that this is no small feat, even for established researchers)
    • Find the partial identification region for each study, using all plausible models and imputations of missing data.
    • Find the intersection of the regions.
  • This intersection gives the set of possible point estimates.

My question:

  • Would this ultimately change the point estimates that much? I suspect it would not.
  • It would make meta-analyses substantially more difficult. It would also raise the quality of them. But is the quality-quantity tradeoff worth it? It is not clear to me. I do not have a strong prior as to whether more but weaker evidence is better than less but stronger evidence. Many decisions in medicine are made on the basis of no evidence at all. These decisions get made every day, regardless of whether evidence exists.  Should doctors be making choices based on flawed evidence or no evidence at all?
  • People well trained in statistics would certainly appreciate the increased uncertainty communicated by reporting an identified set rather than a point estimate with confidence intervals.
  • However, I have trouble picturing any of my medical school classmates (except for the tiny handful also pursuing health economics PhDs) caring about the difference between the two.
  • Most likely, physicians and patients would mentally collapse the result to the midpoint of the set, just to get the point estimate. To be honest, I would, too, if I was trying to communicate to other physicians or to patients about how effective treatment was.
  • And really, what’s the cost-benefit tradeoff of interpreting the results in an overly simplified manner? I suspect that the benefits of slightly oversimplifying outweigh the costs. Medicine is already overwhelmingly cognitively complex. Cognitive mistakes in medicine lead to innumerable injuries and even deaths. Perhaps, as a clinician, I should rather direct my brain space towards preventing avoidable mistakes than thinking about the difference in uncertainty conveyed to patients by point vs set identification.

One final thought: I fully support using better methods whenever possible.  I am just pessimistic about the potential of those statistical nuances to filter down into clinical practice and positively impact real decisions.

MD-PhD in Health Economics Admissions – PhD Departments

This is a continuation of my series on joint degrees in medicine and social sciences, in response to frequently asked questions from undergraduates.

Health economics can be studied in numerous types of PhD departments, with a range of flavors.  Here I have listed some of the key considerations when deciding where to pursue a PhD.  I then discuss how these vary between Economics Department, Public Policy Schools, and Public Health Schools

Key Considerations:

  • Coursework: Coursework will consume a substantial amount of your time as an early PhD student. Those courses should teach skills and subject matter you want to learn.
  • Peer and Faculty Research Interests: Your peers and faculty will shape your research topics and provide critical feedback on your work. Your will have to convince them your research is interesting and robust. Their perspective and research background will profoundly shape your research questions and taste.
  • General Pros and Cons: Some additional considerations to have in mind.

  Continue reading

MD-PhD in Health Economics Admissions – Testing

This is a continuation of my series on joint degrees in medicine and social sciences, in response to frequently asked questions I receive from undergraduates.

The short version: Take the MCAT and the GRE. Do well on the MCAT and GRE quantitative section.

The longer version: Like with coursework, there are no efficiencies or shortcuts here. You have to separately prepare for MD and PhD admissions and earn test score competitive for both.

Continue reading

MD-PhD in Health Economics Admissions – Coursework

This is a continuation of my series on joint degrees in medicine and social sciences, in response to the frequent asked questions I receive from undergraduates.

The coursework requirements to be eligible for a joint program are substantial. The MD and the PhD each have their own set of pre-requisites. And unlike standard MD-PhD programs, there is essentially no overlap between the two sets of courses. For medical school, the pre-requisites are highly official and easy to find information about. For PhDs, the pre-requisites are equally binding, but often not laid out as explicitly.

Continue reading

MD-PhD or MD-Masters? Advice For Undergrads Interested in Health and Social Sciences

This is my first post in a series about joint degrees in medicine and social sciences, in response to frequently asked questions from undergraduates.

Introduction

As an MD-PhD student, I frequently get asked to speak to younger students about graduate school.  Students concerns are often as follows. They know they want to go to medical school and study social sciences. But should they pursue a master’s degree or a PhD in the social sciences? They find it hard to tell the difference (and justifiably – they haven’t pursued either yet).

Continue reading