My talk to Association of Health Care Journalists annual conference in Denver, March 27, 2014.

1. Lessons from reviewing 1,889 stories
Gary Schwitzer
Publisher
HealthNewsReview.org

2. Our criteria: Does the story
explain…
• What’s the total cost?
• How often do benefits occur?
• How often do harms occur?
• How strong is the evidence?
• Are there alternative choices?
• Is the condition exaggerated?
• Is this really a new approach?
• Is it available?
• Who’s promoting this?
• Do they have a financial conflict of interest?
69% 
66% 
65% 
61% 
57% 
Percent
unsatisfactory
after 1,889
story reviews –
7 years

3. Ignore
or
Minimize
Potential Harms

4. Most common flaws
• Conveying a certainty that doesn’t exist
– Exaggerating effect size
– Using causal language to describe observational
studies
– Failing to explain limitations of surrogate
markers/endpoints
– Single source stories with no independent
perspective
– Failing to independently analyze quality of evidence

5. “Bad science is
no excuse for
bad journalism.”

6. Exaggerating effect size –
Absolute vs. Relative Risk
• Two ways of saying the same thing
• One way – relative risk reduction – makes effect size seem
larger.
– Risk of something happening in one group relative to risk in another.
• Other way – absolute risk reduction – makes effect size seem
smaller.
– The chance that something will happen to someone – a ratio of
number of people who have an event divided by all of the people
who could have the event.
• We absolutely think you should use the absolute figures.

7. Tamoxifen cut breast cancer rate
in half

8. Chance of breast cancer over 6 yrs.
Women on placebo – 3.3%
Women on tamoxifen – 1.7%
50% relative risk reduction
1.6% absolute risk reduction
OR – if 100 women took tamoxifen instead of placebo for
6 years, there would be 2 fewer cases of breast cancer.

9. Oh, by the way….
Tamoxifen users were >2 times more likely to develop
uterine cancer or blood clots in lungs/legs.
In relative risk terms, that’s 210% greater risk!
(Absolute risk increased from 0.5% up to 1.1%)
Net effect for every 1,000 women on tamoxifen:
• 16 fewer breast cancers
• 5 more serious blood clots
• 6 more uterine cancers

10. Risk reduction must be stated in
absolute, not just relative, terms
• NBC News reported, “In women, aspirin reduces the rate of stroke
blood clots in the brain, the most common type, by 24% but had
little effect on heart attack risk. In men, the daily aspirin cuts heart
attack risk by 32 percent but had little effect on stroke risk.”
– HealthNewsReview.org comment: “24 percent of what? And 32
percent of what? The journal article upon which the story is
based did state the absolute values for both benefits and risks.
Over 6.5 years:
Daily aspirin is predicted to result in:
- 2 fewer strokes and 2.5 more major bleeds per 1000 women
- 8 fewer heart attacks and 3 more major bleeds per 1000 men.
To many viewers, those numbers will mean more than 32% or 24%.”

11. Last example: absolute vs. relative
3rd graf of NYT story:
“About 8 percent of 2- to 5-year-olds were
obese in 2012, down from 14 percent in
2004.”
From the published study:
“Overall, there have been no significant
changes in obesity prevalence in youth or
adults between 2003-2004 and 2011-2012.”

12. • Results suggest a systematic bias of major
newspapers preferentially covering medical
research with weaker methodology.
• More likely to cover observational studies & less
likely to cover randomized clinical trials
• When they cover observational studies, they
select those of inferior quality.

13. Observational study vs.
randomized clinical trial
• Observational study: observing what happens to people
under exposure conditions that have been self-selected
or determined by influences outside the control of the
researcher. The researcher can choose what exposures
to study, but does not influence them. (Susceptible to
bias and confounding factors)
• Randomized clinical trial: A true experiment. The
researcher controls the exposure and assigns people
randomly to experimental or control group.

14. Using causal language to describe
observational studies
• Observational studies cannot prove cause and
effect so it is inaccurate to use terms like
“benefits…protects…reduces risk”
• These studies can only show a statistical
association, so all you can say is that. We
offer a primer, “Does The Language Fit The
Evidence? – Association Versus Causation.”

15. Association ≠ Causation

16. Stories on:
• Alcohol can make men smarter – CBS
• Eating greater variety of fruit reduces risk of Type 2 diabetes – NYT
• Small reduction in sodium in American diet could save half a million lives/yr. – NYT
• Coffee may protect against heart failure – ABC, CBS, HealthDay, CNN
• The St. Louis Post Dispatch reminded readers:
Researchers caution, however, that they can’t be sure whether these
associations mean that drinking coffee actually makes people live longer.
• Coffee can kill you – Atlanta magazine
• Coffee may cut risk for some cancers – CNN
• Reader comments:
• “I am so f—ing sick of these studies. Stop wasting our time.”
• “Correlation IS NOT causation!!!!”
• “The statistics book in a class I’m taking uses coffee as an example of
statistics run amok.”

17. Surrogate markers/endpoints
• Measuring an outcome, such as a lab test, that substitutes
for measuring an important life event – but may not be an
important finding in itself.
• “A difference to be a difference must make a difference”
– Drug may reduce cholesterol but not prevent MI/death
– Drug may increase bone density but not reduce hip fractures
– Cardiac Arrhythima Suppression Trial (CAST) – drugs to reduce
premature ventricular contractions. Study stopped early
because drugs led to higher death rate and nonfatal problems
than placebo.
– Avastin for breast cancer - drug may extend progression-free
survival but not impact overall

18. Failure to explain limits of surrogate
markers
• USA Today: “New drug ‘may turn back the clock on heart disease.’ ”
– HealthNewsReview.org analysis: “The story focuses excessively on the
‘unprecedented’ changes in cholesterol achieved with the drug. …
these are surrogate markers that may not reflect any real benefit on
the outcomes that matter to patients.”
• WebMD: “Beet Juice Good for Brain: Drinking Beet Juice Increases
Blood Flow to Brain and May Fight Dementia.”
– HealthNewsReview.org analysis: “Surrogate endpoints do not
necessarily reflect on overall health outcomes. And especially not
from a study of 14 people over four days. And from this we get the
headline that ‘drinking beet juice may fight dementia’ ???”
• HealthNewsReview.org linked to 10 stories that framed a tiny short-
term study of a biochemical marker of bone turnover as if it were
definitive advice to women that, for example, “Two glasses of wine
a day ends menopause misery.”

20. Whom to believe? Framing matters.
• Dueling news stories on new prostate cancer
radiation therapy: Breakthrough?
Or not a home run?
– Medical News Today: “breakthrough…extends lives”
vs.
– NY Times Well blog (quoting independent expert): “I
think this is a big deal. It’s not a home run, but it’s a
nice advance.”

21. Whom to believe? Framing matters.
• Dueling news stories on brain wave scan for ADHD.
Important tool? Or waste of money?
– NJ Star-Ledger, Fox News, NY Daily News, Bloomberg – all gushing praise such
as “this should be an important tool” quote from MD in the study
vs.
– NY Times quoted independent expert “skeptical about the test”
– LA Times quoted independent expert “The current scientific research really
doesn’t support (this) as a diagnostic tool. I would caution people.”
– ABC News quoting independent expert: “I don’t know that this going to
help...it’s going to make people spend money needlessly. They can charge for
it and it gives you a pseudo-scientific basis for diagnosis.”

22. Whom to believe? Framing matters.
Dueling news stories on melanoma detection device.
Helping doctors? Or causing dissent/polarizing the field?
• Cleveland Plain Dealer: “helps doctors detect deadly cancer”
• WTVF Nashville & KFSN Fresno – among the stations using a syndicated news
service’s story that claimed “the Cleveland Clinic named the device one of its top
medical innovations of 2013.”
• KDVR Denver allowed a doctor to call it the biggest advance in melanoma in his
26-year career
vs.
• NY Times quoted 3 independent experts:
1. “This should still be considered to be in the developmental stage.”
2. A dermatologist on the FDA panel who voted against approval
3. Biostatistician: “(The test) just says everything is positive. I don’t think it
helps an aggressive doctor and unaggressive doctors could do just as well if
they were more diligent without the device.”

23. Whom to believe? Framing matters.
• Dueling news stories on meds for mild-moderate depression.
Give pause about prescribing? Or no reason to avoid meds?
– HealthDay: “Severely depressed gain most from antidepressants: But
that’s no reason to avoid meds for moderate cases.”
vs.
– LA Times: “Study finds medication of little help to patients with mild,
moderate depression”
– USA Today (quoting independent expert): Such findings “demonstrate
a failure in the system: These drugs were not thoroughly tested in mild
to moderate depression prior to their approval.”

24. Whom to believe? Framing matters.
• Dueling news stories on Alzheimer’s drug. BY
THE SAME NEWS SOURCE! Reverses disease in
mice? Or a flop?
– MedicalDaily.com: “Cancer drug may help reverse
Alzheimer’s.”
AND
– “Targretin a flop: Potential Alzheimer’s drug fails
retrial.”

25. Whom to believe? Framing matters.
• Dueling news stories on limitations of colonoscopy.
Miss many cancers? Or concerns overblown?
– NY Times: “Colonoscopies miss many cancers, study finds.”
VS.
– ABC News: “The Case for Keeping Colonoscopy: Cancer Experts
Say Concerns Over Colonoscopy Effectiveness May Be
Overblown.”
– CBS News’ Katie Couric: “Don’t use this study as an excuse not
to get screened.”

26. Whom to believe? Framing matters.
• Dueling news stories on prostate cancer screening study.
Reduces deaths? Or isn’t saving lives/value questioned?
– MSNBC: “Regular prostate screening reduces deaths”
– HealthDay: “PSA testing cuts cancer death risk”
– Bloomberg: “PSA screening reduces deaths”
vs.
– Associated Press: “prostate cancer screening isn’t saving lives,
study finds”
– CNN.com: “Value of mass prostate cancer screenings
questioned.”

27. The final line of the NEJM article:
More information on the balance of
benefits and adverse effects, as well as
the cost-effectiveness, of prostate-
cancer screening is needed before
general recommendations can be made.
Maybe that should have received a little
more attention.

28. Framing
• Star Tribune: “Boy, 12, dies after historic
transplant to treat HIV, leukemia.”
– “He was in line to become the second person in
the world to be cured of both deadly illnesses by
the extraordinary type of bone marrow transplant,
doctors said.”
– Maybe. But in fact he was in line to die from a
transplant that was not historic because it did not
treat his HIV & leukemia.

29. The tyranny of the anecdote
• Stories may use only positive, glowing patient
anecdotes but fail to capture trial dropouts,
compliance problems, patient dissatisfaction or
the choice to pursue less aggressive options.
• In one example, four major US news sources all
used the same “breakthrough” patient anecdote
of one man’s reported improvement from a heart
failure stem cell experiment.
– This suggests another common problem: public
relations people spoon-feed journalists the patient
anecdotes that put an intervention in the most
positive light.

30. Single source stories & journalism via
news release are unacceptable
• ABC’s Good Morning America asked, “Could a cure for obesity be just a pill away?”
HealthNewsReview.org analyzed the sourcing in this story:
– “In a bizarre twist, the reporter used the potential conflicts of interest of the doctor he
interviewed in the studio as evidence of his expertise. He noted that the doctor is a
consultant to the manufacturer of the pill and then said, “You’re the right man to talk
to.” Really? In a recent journal publication, this researcher listed financial ties to 17 drug
and device makers, including the maker of the (pill in question)… viewers were not
clearly told that all of the results reported in the story came from a trial controlled by
the company that has not been independently reviewed.”
• The Wall Street Journal reported, “Depression treatment Cymbalta effective as pain reliever,
Lilly says.” HealthNewsReview.org wrote:
– “This 189-word story failed on almost every count:
– no independent perspective;
– a quote apparently lifted from a company news release;
– The company news release actually did a better job of explaining the study than did this
news story.”

32. Stories about screening tests
• Sir Muir Gray: “All screening programmes do
harm; some do good as well.”
• All screening stories should explain the
tradeoffs of potential benefits and potential
harms

33. Rates of new diagnosis and death for five types of cancer in the US, 1975-2005.
Moynihan R et al. BMJ 2012;344:bmj.e3502
©2012 by British Medical Journal Publishing Group

34. Unquestioning news coverage of
disease awareness campaigns

35. Movember misinformation in Chicago Sun-Times from
star of “In the Bedroom with Dr. Laura Berman”
• She wrote that “men aren’t aware of the warning signs of prostate cancer” and
went on to list some, including “frequent lower back pain” and “swelling of the
legs and feet and weight gain.” But there are no warning signs that are specific for
prostate cancer. Her message could create unnecessary anxiety and lead to
unnecessary testing.
• She mentioned an industry-funded survey suggesting that men “greatly
underestimate their risk of prostate cancer.” She did not provide a reference for
that survey. But peer-reviewed published work has shown that men vastly
overestimated such risk.
• She also wrote as a statement of fact that “more frequent ejaculation could help
men to lower their risk of prostate cancer.” At best, the research on this point is
inconclusive, with other studies suggesting that men who have more ejaculations
are more likely to develop prostate cancer.
• She may know something about ejaculation but she’s shooting blanks on this one.

36. Fawning coverage of new technologies
(proton beam therapy & robotic surgery)
• KERA public radio in Dallas reported on both
technologies in its “Battlefield Breakthrough” series.
– Headline: “Zapping Cancer With A Ray-Gun.” We wrote:
“The story presents not one word of caveats, concerns or
limitations. There was no independent source, no data
about efficacy.” Instead, there was fascination: “looks like
something out of Star Wars.”
– Headline: “Robots Slice Time Under The Knife.”
We wrote: “Star Wars was a theme in that one as well. In fact, the
formula was the same as for part one with proton – more battlefield
breakthroughs. … Maybe we need to wait a bit to see how this arms
race plays out with evidence and data before proclaiming battlefield
breakthroughs just yet.”

37. Going easy on health business stories
• Reuters
– Headline: Medtronic tests stents for erectile dysfunction. Our comment: “An initial feasibility study
in 30 men is worthy of news coverage? With a company-sponsored investigator as the only source?
Not in our view.”
• Chicago Tribune
– Headline: Area firm’s pneumonia treatment passes test. Our comment: “It features the company
president’s positive assessment of the study results but does not seek independent comment. … It
fails to make clear that the results of the trial have not been peer-reviewed or published and that
results of a related trial have not been released at all.” 56
• New York Times
– Headline: “Test of Eye Drug Is Said To Show Success in Elderly.” Our comment: “The piece …
should’ve slapped a bigger warning label on this cocktail of leaked summaries of secret results,
speculation, and an inadequate dash of independent evaluation.”
• Wall Street Journal
– Headline: “AstraZeneca Says Brilinta Beats Plavix in Clinical Trial.” Our comment: “… lets a drug
company get away with making superiority claims without releasing data. … All information in the
story appears to come from company sources.”
• Los Angeles Times
– Headline: “Drug for menstrual cramps in the works.” Our comment: “This story reports that data
were presented at a scientific meeting. But it didn’t tell readers what the data showed. But it did
allow a company VP – the only person quoted – to say this could be a ‘breakthrough.’ Wow.”
• Minneapolis Star Tribune
– HealthNewsReview.org found four stories within six weeks in which the Star Tribune established a
clear pattern of writing nice things about local medical device industries, but did not lead in
reporting negative stories (e.g., on troubling Medtronic stories in their own back yard, often scooped
by Milwaukee Journal-Sentinel – in another state, a smaller paper.)

38. Blog roll:
• Dr. Richard Lehman’s weekly journal review on
a BMJ blog.
– Funny. Hard-nosed. A treasure.
• Retraction Watch/Embargo Watch.
• Knight Science Journalism Tracker
• Skeptical Scalpel blog.
• Not Running A Hospital blog.

39. 1. Not all studies are equal. All have flaws. You
and your audience should appreciate the
limitations inherent in any study design.
2. If you rely on medical journals as a main source
of your news, you are getting and giving an
imbalanced view of new treatments, tests,
products and procedures.
3. Both industry interests and journal policies
tend to promote positive findings.
4. If you rely on presentations at scientific
meetings as a main source of your news, you may
be promoting claims that have not been validated
by any independent expert sources.
5. There are a growing number of resources that
can help you evaluate claims and evaluate the
underlying science in journal articles,
presentations at scientific meetings, news
releases and interviews with experts.

40. “Good journalism has a subtle feature of
reticence. We don’t publish everything we hear.
We filter. We curate…and this requires the
willingness to not publish things that are
unlikely to be true.”
Joel Achenbach, Washington Post
“The Shroud of Turin, pseudoscience and journalism”

41. 8th straight AHCJ
workshop discussing
HealthNewsReview.org
– could be our last

44. Who’s going to continue this work?

45. @Thank you
gary@healthnewsreview.org