For the 4th time in 5 years, I led a discussion at Knight Science Journalism Program at MIT's Medical Evidence Bootcamp.

1. Holding a mirror up to two industries
We were awarded a Mirror Award – hosted by Syracuse U.
Newhouse School of Communications for those “who hold a
mirror to their own industry for the public’s benefit.”

2. Lessons to improve our reporting on evidence:
An analysis of 1,800+ stories over 7 years
Gary Schwitzer
Publisher
MIT Medical Evidence Boot Camp
December 4, 2012

3. Infoxication

4. What others are saying about us
The editors of the journal PLoS Medicine wrote: “Schwitzer’s alarming report card of the
trouble with medical news stories is thus a wake-up call for all of us involved in
disseminating health research-researchers, academic institutions, journal editors,
reporters, and media organizations-to work collaboratively to improve the standards of
health reporting.”
Susan Perry, on her MinnPost.com column referred to HealthNewsReview.org as:
“indispensable to consumers & journalists”
Dan Gillmor, director of the Knight Center for Digital Media Entrepreneurship at Arizona
State University’s Walter Cronkite School of Journalism and Mass Communication: “Calm
and thorough analysis of health news journalism from HealthNewsReview.org”
Dr. Steven Kussin, Director of The Shared Decision Center, wrote: “All doctors should
suggest patients register at HealthNewsReview.org.”
Science journalist Paul Raeburn on the Knight Science Journalism Tracker:
“I’m beginning to think that Schwitzer’s criteria for judging stories ought to be printed on
wallet cards for reporters, like Miranda warnings, to remind them what questions to ask. I
could use one of those myself.”

5. Our criteria: Does the story explain…
• What’s the total cost?
• How often do benefits occur?
• How often do harms occur?
• How strong is the evidence?
• Is the condition exaggerated?
• Is this really a new approach?
• Is it available?
• Are there alternative choices?
• Who’s promoting this?
• Do they have a financial conflict of interest?

8. After 1,800 stories
over 6+ years
~70% of stories fail to:
✔ Discuss costs
✔Quantify potential benefits
✔ Quantify potential harms
✔ Evaluate the quality of the evidence

9. Ignore
or
Minimize
Potential Harms

10. In an election year, when we should have
spent more time helping citizens weigh
why we spend 17% of GDP on health
care, yet leave 17% of our neighbors
uninsured….

11. …instead, news often painted a
kid-in-candy-store picture
of U.S. health care interventions
 Terrific
 Risk-free
 Without a price tag

13. What CNN didn’t report…
• Resignations, questions about conflicts of
interest and fraud probes of the program in
question.
• Not likely to be the kinds of angles and issues
one pursues when there are “exclusive”
reporting arrangements between a news
organization and a medical center.

14. Robert Bazell: “For the first time ever an
experimental drug is showing great promise
of slowing the progression of Alzheimer's
disease.” NBC News - October 8, 2012
FIRST TIME EVER?
July 17, 2012: New Alzheimer’s Drug Slowing Progression of the Disease – CBS
March 8, 2012: Alzheimer’s treatment in late stages of disease does slow progression
(The Independent)
October 20, 2011: Antiviral Drugs May Slow Alzheimer's Progression (Science Daily)
May 11, 2009: Can New Drug Slow Progress of Alzheimer's? (ThirdAge.com)
August 3, 2008: Drugs May Slow Progress Of Alzheimer’s (NPR on two drugs)
5-minute Google search =
6 different approaches reported to slow progression of AD just in past 4 years

15. Questions you might ask:
“a revised look at two trials that were declared failures” – What does that mean?
How big was potential benefit? AP reported:
“Taken separately, the studies on the drug - Eli Lilly & Co.'s solanezumab - missed their
main goals of significantly slowing the mind-robbing disease or improving activities of
daily living. But pooled results found 34 percent less mental decline in mild
Alzheimer's patients compared to those on a fake treatment for 18 months.”
How big were potential harms? Angina…brain swelling…brain bleeds

16. Common flaws: too much stenography – not
independent vetting of studies in journals
Deifying big names/big journals - Publication in a medical
journal does not guarantee the findings are true (or even
important).
Not ready for prime time – journals meant for conversation
among scientists
Never intended to be sources of daily news. So if you’re going to
use them that way, you simply must be aware of the
landscape:
• retractions, research fraud, fabrication, falsification of
data
• unpublished data (BMJ special edition on “the extent,
causes and consequences of unpublished evidence”)
• ghostwriting of journal articles (The Public Library of
Science hosts a “Ghostwriting Collection” on its website.)

17. Common flaws (Examples provided later upon request)
Failure to evaluate inherently weak science
Idolatry of the surrogate – Not understanding or simply not
reporting that surrogate outcomes (like tumor shrinkage) may not
translate into clinically meaningful outcomes (longer life).
Using causal language to describe observational studies
Exaggerating effect size - Using relative, not absolute risk
reduction numbers.
Reckless extrapolation - Predicting what may happen in humans –
and soon - based on very preliminary animal / lab science.
Lack of awareness of conflicts of interest & other ethical issues
• commercialization of research: contract research organizations,
commercial IRBs or institutional review boards, medical
education and communication companies

19. Misreporting of Observational Studies:
Can Scientific Journals Help? (more on journals later)
NBC: “The science that just might justify an American addiction…Bona fide science. A real
study of a lot of people. …There are benefits… may decrease heart disease by as much as
37 percent.”
The piece ended by quoting Mark Twain:
“The only way to keep your health is to eat what you don’t want, drink what you don’t like,
and do what you’d rather not.”
We cited another Mark Twain quote – just substitute “network TV news” for “newspaper”
– imagine if Twain had lived to see today’s TV news:
“If you don’t read the newspaper, you’re uninformed. If you read the newspaper, you’re
mis-informed.”

20. Two birds killed with one stone in HealthDay story
1. Over-reliance on news releases
2. Idolatry of the surrogate

21. Press release: ”In recent years, drugs that inhibit BRAF activity have rapidly halted and
reversed tumor growth in about 90 percent of treated patients, but most patients’
response is temporary, with tumor growth resuming in six or seven months.
Investigations into how this resistance emerges have suggested that the MAPK pathway
gets turned back on through activation of MEK, another protein further down the
pathway.”
Story: “Drugs that inhibit BRAF activity can rapidly stop and reverse tumor growth in
about 90 percent of patients. But the response is temporary in most cases, and tumor
growth resumes in six or seven months, the researchers explained. Previous research
suggested that this drug resistance develops because the MAPK pathway gets turned
back on through activation of MEK, another protein that is part off the MAPK pathway.”
---------------------------------------------
Quality of the evidence?
The story dutifully reports on the primary outcome of the study, progression-free
survival, but doesn’t explain that this may not translate to an overall increase in survival
time for patients taking the drug combination.

22. Our review:
The story fails to point out the intermediate endpoints that look at risk factors for heart
attack and stroke (such as cholesterol, and signs of thickening or calcium deposits in
arteries) are not what we care about, which is whether heart attacks and strokes were
any more or less common. Since it takes longer to detect such events, the researchers
chose to look at markers for future disease rather than cardiovascular events. That’s a
legitimate way to explore a hypothesis, but must be emphasized as preliminary. Don’t
forget that we got into the widespread but misguided practice of prescribing estrogen
for cardiovascular prevention back in the 80s based on just such assumptions, based on
data that showed HT had favorable effects on cholesterol and other markers–yet in the
end it caused more heart attacks and strokes.

23. “The numbers don’t lie….or explain
anything about what they mean.”

24. Screening Madness
• Crusading
• one-sided
• advocacy

25. I am not anti-screening
• I do oppose imbalanced, incomplete messages
on screening tests
• I do oppose messages that emphasize benefits
and minimize or ignore harms
• I do oppose messages that only tell anecdotes
of those who claim their life was saved by
screening – something that can never be
proven.

26. Crossing line from journalism to advocacy

27. A physician wrote to me after seeing
this:
"Could a political reporter say
'Vote for Obama?'!"

28. Crusading advocacy on CBS News
“Well, we’ve had a conclusion for many, many years at Susan G. Komen,
almost a generation. Screening saves lives. The 5-year survival rates for breast
cancer diagnosed early is 98 percent…and this is largely due to screening and
early diagnosis.”
Left CBS, left “journalism,” just joined ABC

29. Can you imagine a political
reporter wearing a button saying,
“Vote for Romney” ?

30. Crossing line from journalism to advocacy
"For you women out there and for the men who
love you, screening matters. Do it. This disease
can be completely curable if you find it at the
right time."
Katherine O’Brien has metastatic breast
cancer – on her ihatebreastcancer blog:
“Early detection is not a cure. ‘Completely
curable’ is a like a fat man wearing a hockey
jersey. It covers a lot of ground.”

31. This is what he wrote:
The question was simple enough.
"Do you want to be scanned?" asked the hospital PR woman.
Um, sure?
(More on Minneapolis Star Tribune later)

32. “All screening programmes do harm;
some do good as well.”
- Dr. Muir Gray
Some screening stories we should hear more about….

34. Track this yourself:
• When you disagree with evidence, use politics to attack the process
• There has been public ridicule of fact that a woman pediatrician heads USPSTF.
• Since when does it require subspecialty in urology or radiology to evaluate evidence?

35. Leaders of ZERO: The Project to End Prostate Cancer
When the US Preventive Services Task Force recommended against routine screening of
men with the PSA blood test, CEO Skip Lockwood said the UPSTF’s advice “condemns
tens of thousands of men to die this year and every year going forward…”
COO Jamie Bearse wrote about the chief medical officer of the American Cancer Society:
“Otis Brawley has killed more men by giving them an excuse to not be tested.”
Track this yourselves: Why is it that evidence-based, shared
decision-making advocates call for choice but screen-everybody
advocates promote mandate not choice?

36. Brawley on offensive:
“Many of these free screenings are designed more to get
patients for hospitals and clinics and doctors than they are to
benefit the patients. That's a huge ethical issue that needs to be
addressed.
We're not against prostate cancer screening. We're against a man
being duped and deceived into getting prostate cancer
screening."

37. Duped & Deceived?
Tim Glynn, lawyer, age 47
when doctor “decided I’d have
a PSA test without consulting
me.”
“Men should be aware of the truly
Profiled in Shannon Brownlee’s
terrible consequences. As a NYT Sunday mag piece,
screening tool, you could do as “Can Cancer Ever Be Ignored?”
well by throwing dice on a table.”

38. DECISIONS Study: PSA Decision
• 70% of men reported a discussion before PSA test
• MD generally initiated discussion (65%)
• MD generally recommended screening (73%)
• “Pros” discussed some/a lot in 71% of discussions,
“cons” not discussed at all in 68%
• 55% reported being asked their opinion
• 48% couldn’t answer any of three knowledge
questions
Hoffman, et al. Arch Intern Med 2009; 169:1611

39. Prizes for Prostates

40. The marketing of screening
In 2010 after the National Lung Screening Trial
results were released, the ACS posted on its blog:
"It's only been a few days since researchers
released preliminary results...(which) although
enormously promising…not enough to call for
routine use of this screening test, even in heavy
smokers. ...But our greatest fear was that forces
with an economic interest in the test would
sidestep the scientific process and use the
release of the data to start promoting CT scans.
Frankly, even we are surprised how quickly that
has happened."

41. TheNNT.com: number needed to treat, to screen, to harm

42. • Began with virtual colonoscopy –after routine physical.
• But more than just colon is captured & CT showed something on kidney, liver, and lungs.
• Kidney and liver issues benign after liver biopsy, PET scan and more CT scans.
But lung questions led to major lung surgery, sawing him open through ribs
“I awoke in the recovery room after 5 hours, with a chest tube, a Foley catheter, a
subclavian central venous catheter, a nasal oxygen catheter, an epidural catheter, an arterial
catheter, subcutaneously administered heparin, a constant infusion of prophylactic
antibiotics, and patient-controlled analgesia with intravenously administered
narcotics…..Excruciating pain.”
• No malignant disease – all “incidentalomas”
• Total cost > $50,000
• All precipitated by a screening test

43. Sharon Begley in Newsweek:
Dr. Stephen Smith, Professor emeritus of
family medicine at Brown University
School of Medicine, tells his physician not
to order a PSA blood test for prostate
cancer or an annual electrocardiogram to
screen for heart irregularities, since
neither test has been shown to save lives.
Rather, both tests frequently find
innocuous quirks that can lead to a Dr. Rita Redberg, professor of medicine at
dangerous odyssey of tests and the University of California, San Francisco,
procedures. and editor of the prestigious Archives of
Internal Medicine, has no intention of
having a screening mammogram even
though her 50th birthday has come and
gone. That’s the age at which women are
advised to get one. But, says Redberg, they
detect too many false positives (suspicious
spots that turn out, upon biopsy, to be
nothing) and tumors that might regress on
their own, and there is little if any evidence
that they save lives.

44. • Reported on woman with DCIS – ductal carcinoma in situ
• Reported on her choice to pursue active surveillance rather
than immediate aggressive intervention such as bilateral
prophylactic mastectomy
• Reported on the shared decision-making program at UCSF that
helped her understand the tradeoffs.
THAT IS GOOD JOURNALISM

46. Home-cooking on health biz stories
• Minneapolis Star Tribune lags
behind smaller paper, smaller
market on MSP medical device co.
troublesome issues
• Apparent cheerleading on local
Medtronic, St. Jude Medical,
Boston Scientific stories.
• Search “Star Tribune” on our site
to see 4 story examples in a month
• They’re not alone

47. Journalists could help people understand
and deal with the clash between:
• Science  Intuition
• Evidence  Emotion
• Data  Anecdote
• Recommendations for  Decision-making by an
entire population individual
• What we can prove  What we believe, wish, or hope
• Grasping uncertainty and  Promoting false certainty
helping people apply where it does not exist
critical thinking to
decision-making issues

48. • Surrogate markers may not tell the whole story
• Does The Language Fit The Evidence? – Association Versus Causation
• Resources for Reporting on Costs of Medical Interventions
• “Off-label” Drug Use and Marketing
• 7 Words (and more) You Shouldn’t Use in Medical News
• Problems with Reporting on News from Scientific Meetings
• Absolute vs. Relative Risk
• Number Needed to Treat
• Commercialism
• Single Source Stories
• FDA Approval Not Guaranteed
• Phases of Drug Trials
• Medical Devices
• Animal & Lab Studies

51. Read John Ioannidis to learn pitfalls
of a steady diet of journal stories
PLoS Med 2005; 2(8): e124

52. Journals complicit in miscommunication
• Editors of the HEART Group journals last week
stated that “inappropriate word choice to describe
results can lead to scientific inaccuracy.”
– J AM COLL CARDIOL, Vol. 60, No. 23, 2012
• “Are we making a mountain out of a mole hill? A
call to appropriate interpretation of clinical trials
and population-based studies”
– Am J Obstet Gynecol, published online 11/29/12
• “Spin and Boasting in Research Articles.”
- Commentary in Arch Pediatr Adolesc Med:
[published online October 2012]

53. “I honestly believe it is better to know
nothing than to know what ain’t so.”
JOSH BILLINGS (PEN NAME OF HUMORIST
HENRY WHEELER SHAW, 1818 – 1885)

54. Ask yourselves
• If your beat includes covering
studies/clinical news, what is mix
of stories just about “new stuff”?
• Do you think you might be
reporting too much of this?
• If so, have you spoken with
editors about this? Do you want
our help?
• If you don’t think you report too
much of this, how would you
stack up on our 10 criteria?
• If you don’t accept or agree with
our 10 criteria, what criteria do
you use?

55. Thank you
GGary@HealthNewsReview.org