My workshop talk at Association of Health Care Journalists 2013 conference in Boston.

1. Birds, Bears, Turtles, Trains,
Coupons, Cocks…and Moses
Gary Schwitzer
Publisher
Association of Health Care Journalists
Boston – March 14, 2013

2. We review stories
that include claims
about…
• Medical treatments
• Tests
• Products
• Procedures

4. Our criteria: Does the story explain…
• What’s the total cost?
• How often do benefits occur?
• How often do harms occur?
• How strong is the evidence?
• Is the condition exaggerated?
• Is this really a new approach?
• Is it available?
• Are there alternative choices?
• Who’s promoting this?
• Do they have a financial conflict of interest?

5. After 1,800 stories
over 7 years
~70% of stories fail to:
✔ Discuss costs
✔Quantify potential benefits
✔ Quantify potential harms
✔ Evaluate the quality of the evidence

6. News stories often paint a
kid-in-candy-store picture
of U.S. health care
 Terrific
 Risk-free
 Without a price tag

8. Ignore
or
Minimize
Potential Harms

9. The award for
worst news
coverage:
THE WINNER IS:
SCREENING
TESTS
(often involving this
gland)

10. Turtles, birds, bears
• Turtles move too slowly to ever be
dangerous and don’t need treatment.
• Birds are so fast you’ll never catch
them. Too late to try to treat.
• Bears are dangerous, but move slowly
enough that you can catch them.
• Screening tests can only make a difference for the
bears.

11. Lead-time bias Beth gets on in Boston
Nancy boards in NY
Washington
Train crashes in Washington. Both die.
It may look like Beth was a longer-term
survivor of the train ride, but only
because she got on earlier than Nancy.

12. While running for president Rudy Giuliani announced in a 2007 campaign ad:
“I had prostate cancer, 5, 6 years ago. My chance of surviving prostate cancer—and
thank God, I was cured of it—in the United States? Eighty-two percent. My chance
of surviving prostate cancer in England? Only 44 percent under socialized
medicine.”
Yet despite this impressive difference in the five year survival rate, the mortality
rate was about the same in the US and the UK.
How can that be? Remember the train? Lead-time bias and overdiagnosis. Far more US
men get PSA test than men in the UK, contributing to US’ higher survival rate.

13. Susan G Komen for the Cure’s mammography advertisement during breast cancer awareness
month, 2011.
Woloshin S , and Schwartz L M BMJ 2012;345:bmj.e5132
©2012 by British Medical Journal Publishing Group

14. Common flaws: too much stenography – not
independent vetting of studies in journals
Glorifying big names/big journals - Publication in a medical
journal does not guarantee the findings are true (or even
important).
Not ready for prime time – journals meant for conversation
among scientists
Never intended to be sources of daily news. So if you’re going to
use them that way, you simply must be aware of the
landscape:
• retractions, research fraud, fabrication, falsification of
data
• unpublished data (BMJ special edition on “the extent,
causes and consequences of unpublished evidence”)
• ghostwriting of journal articles (The Public Library of
Science hosts a “Ghostwriting Collection” on its website.)

15. Common flaws
Failure to evaluate inherently weak science
Idolatry of the surrogate – Not understanding or simply not
reporting that surrogate outcomes (like tumor shrinkage) may not
translate into clinically meaningful outcomes (longer life).
Reckless extrapolation - Predicting what may happen in humans –
and soon - based on very preliminary animal / lab science.
Lack of awareness of conflicts of interest & other ethical issues
Going soft on business stories or on local stories

17. What CNN didn’t report…
• Resignations, questions about conflicts of
interest and fraud probes of the program in
question.
• Not likely to be the kinds of angles and issues
one pursues when there are “exclusive”
reporting arrangements between a news
organization and a medical center.

18. Prostate cancer scan advance helps Mayo Clinic doctors with early
detection
Minnesota Public Radio
“The scan allows men with cancer to receive treatment that is both faster and
potentially more effective than current tests.”

19. Stories about tests should explain
sensitivity and specificity
• Sensitivity = how many true positives
• Specificity = how many true negatives
• What the Mayo story didn’t report was that
studies have shown false positive tests of 15-47%
- meaning biopsies and more risks.
• And one Mayo report noted 11% false negatives
• Shouldn’t that be reported alongside “The scan
allows men with cancer to receive treatment that
is both faster and potentially more effective than
current tests.”

20. Evidence – not just excitement – about
expensive new technologies

21. Using causal language to describe
observational studies
• Observational studies cannot prove cause and
effect so it is inaccurate to use terms like
“benefits…protects…reduces risk”
• These studies can only show a statistical
association, so all you can say is that. We
offer a primer, “Does The Language Fit The
Evidence? – Association Versus Causation.”

22. “Relative finger length could be used as a
simple test for prostate cancer risk”
Based on what?
The men were shown pictures of different finger length patterns and asked to
identify the one most similar to their own.
Men whose index and ring fingers were the same length -- about 19 percent
of those studied -- had a similar prostate cancer risk, but men whose index
fingers were longer than their ring finger were 33 percent less likely to have
prostate cancer.

23. Association ≠ Causation

24. Exaggerating effect size –
Absolute vs. Relative Risk
• Two ways of saying the same thing
• One way – relative risk reduction – makes effect size
seem larger
• Other way – absolute risk reduction – makes effect
size seem smaller.
• We absolutely think you should use the absolute
figures.

25. Nolvadex (tamoxifen)
Reducing breast cancer risk by 50 .

26. For the first time, there is a clinically proven way for
many women at high risk of developing breast
cancer to significantly reduce that risk.
The proof? In a landmark study…women who took
Nolvadex had 48% fewer breast cancers than
women taking sugar pills.

27. Women who took Nolvadex had 48% fewer breast
cancers….
48% of what ???

28. It's like a coupon…
Extremely Fancy Store What if selected items were..
TVs, washing machines?
48
On selected items!
%
OFF save $100s
Things like candy?
save pennies
“48% of what” matters!
Know the REGULAR price!

29. What is the coupon really worth?
Chance of getting breast cancer over 6 years
Placebo:The REGULAR price!
PLACEBO Nolvadex: The SALES price!
NOLVADEX
3.3% 1.7%
How much do you save?
Absolute risk Savings = Regular price – Sales price
reduction 3.3% 1.7% = 1.6%
Chance of getting breast cancer (over 6 years) with for 6 years,
If 100 women took NOLVADEX instead of placebo NOLVADEX
there would be about 2 lower than with breast cancer.
was 1.6 % points fewer cases of placebo

30. What is the effect of Nolvadex?
How good is the sale?
PLACEBO NOLVADEX
3.3% 1.7%
Chance of outcome (intervention)
Relative Risk =
Chance of outcome (control)

31. Describing the effect of NOLVADEX
So finally....this is how you get to
PLACEBO NOLVADEX
the48% off sale!
3.3% 1.7%
It's the relative risk reduction
1.7%
???
Relative Risk = = 0.52
???
3.3%
"% Lower" format =1 - RR =1 - .52 =.48
At 6 years, the chance of breast cancer for women taking
NOLVADEX was 48 % lower than that of women taking placebo.

32. Two ways of saying the same thing:
the benefit of NOLVADEX
Extremely Fancy Store Extremely Fancy Store
One feels big One feels small
48 %
OFF
1.6%
How you say it matters!
On selected items!
SAVINGS
On selected items!
"Framing"

33. Relative vs. "% off" risk
absolute "savings"
reductions
Chance of death at 1 year Risk reduction
Relative Absolute
Placebo DRUG (1-[DRUG/Placebo]) (Placebo-DRUG)
30% 10% 67%
?? 20%
??
3% 1% 67% 2%
0.003% 0.001% 67% 0.002%

34. The proof? In a landmark study…women
Benefit who took Nolvadex had 48% fewer breast
cancers than women taking sugar pills.
Harm
"Nolvadex isn't for every woman…In the study women taking
Nolvadex were 2 to 3 times more likely to develop
uterine cancer or blood clots in the lung and legs, although each
occurred in less than 1% of women".
”Strokes, cataracts more common with
No numbers
Nolvadex. Most women experience some level
of hot flashes and vaginal discharge".

35. Present benefits and harms the same way

36. The proof? In a landmark study…women
who took Nolvadex had 48% fewer breast
cancers than women taking sugar pills.
210% more uterine cancer and potentially
210% more
life threatening blood clots in the lung and legs.

37. Over the next 6 years, what happened …
PLACEBO NOLVADEX
Benefits: Nolvadex lowered chance
Getting breast cancer 3.3% 1.7%
Harms: Nolvadex increased chance
Having a serious blood clot 0.5% 1.0%
Getting uterine cancer 0.5% 1.1%
Net effect of Nolvadex for every 1000 women:
16 fewer women get breast cancer
5 more women get serious blood clots
6 more get uterine cancer

38. • Surrogate markers may not tell the whole story
• Does The Language Fit The Evidence? – Association Versus
Causation
• 7 Words (and more) You Shouldn’t Use in Medical News
• Problems with Reporting on News from Scientific Meetings
• Absolute vs. Relative Risk
• Number Needed to Treat (more on this in the afternoon)
• Commercialism
• Single Source Stories
• Phases of Drug Trials
• Medical Devices
• Animal & Lab Studies

39. Progression free survival = The length of time during and after the treatment of
cancer, that a patient lives with the disease but it does not get worse.
Versus
Overall survival = do people live longer?

42. Read John Ioannidis to learn pitfalls
of a steady diet of journal stories
PLoS Med 2005; 2(8): e124

43. Journals complicit in miscommunication
• Editors of the HEART Group journals recently
stated that “inappropriate word choice to describe
results can lead to scientific inaccuracy.”
– J AM COLL CARDIOL, Vol. 60, No. 23, 2012
• “Are we making a mountain out of a mole hill? A
call to appropriate interpretation of clinical trials
and population-based studies”
– Am J Obstet Gynecol, published online 11/29/12
• “Spin and Boasting in Research Articles.”
- Commentary in Arch Pediatr Adolesc Med:
[published online October 2012]

44. Annals of Oncology: “Bias in reporting of end
points of efficacy and toxicity in randomized,
clinical trials for women with breast cancer”
(published online January 9, 2013)
Reuters: “Doctors relying on studies published in
top journals for guidance about how to treat
women with breast cancer may not be getting
the most accurate information.”

45. What the authors point out:
• “Investigators go overboard to make their
studies look positive.”
• In 2/3 of studies, that meant not listing
serious side effects in the abstract – which is
all many may read
• In 1/3 of studies, if the treatment didn’t work
as hoped, researchers reported results the
study was not designed to test – “secondary
endpoints”

46. September 2012
Positive “spin” was identified in about half of press releases and
news stories. The main factor associated with “spin” in press
releases was the presence of “spin” in the journal article abstract
conclusion.
In other words, a direct link from
published study news release news story.
Where is the reader left behind in this food chain?

47. Public Misunderstanding of Cancer Risk
• Studies have consistently shown people
overestimate their own risk of cancer
• Other studies show people overestimate risk
factors that have not been proven and
underestimate risk factors that are well-
established

48. “Unrealistic Optimism in Early-Phase Oncology
Trials”
People tended to overestimate the benefits of
the trial they were enrolled in and
underestimate its risks.
-- IRB: Ethics & Human Research 2011

49. We hold a mirror up to journalists – Ask yourselves:
• If you cover studies/clinical
news, how many of those stories
are just about new
treatments, tests, products or
procedures?
• Do you think you might be
reporting too much of this?
• If so, have you spoken with
editors about this? Do you want
our help?
• If you don’t think you report too
much of this, what kind of grades
would you get on our 10 criteria?
• If you don’t accept or agree with
our 10 criteria, what criteria do
you use?

50. Drowning from a firehose of infoxication

51. @garyschwitzer
@ivanoransky

52. Thank you for your attention
Thanks to Drs. Steve Woloshin & Lisa
Schwartz of Dartmouth Medical School for
slides on absolute vs. relative risk
My email:
Gary@HealthNewsReview.org