This document summarizes lessons from analyzing 1,600 health news stories over 6 years. It finds that most stories fail to adequately discuss costs, quantify benefits and harms, evaluate evidence quality, and compare new options to existing ones. There is also a tendency to emphasize benefits and minimize harms. Screening stories in particular are often one-sided and promote false certainty. Overall, health journalism could be improved by providing more balanced and evidence-based reporting.

1. Lessons to improve our reporting on evidence
– analysis of 1,600 stories over 6 years
MIT Medical Evidence Boot Camp
December 8, 2011
Gary Schwitzer
Publisher
HealthNewsReview.org

6. Site stats
 28 reviewers: 19 clinicians or researchers, 7
journalists, 2 breast cancer survivors
 Site launched April 2006
 1,655 stories reviewed
 Blog launched in 2004 - >2,500 posts
 Sole funding from the Foundation for Informed
Medical Decision Making here in Boston
(Co-founders Jack Wennberg, Al Mulley, etc.)

8. After 5.5 years and 1665 stories
 71% fail to adequately discuss costs.
 66% fail to quantify benefit - often how small is
potential benefit
 65% fail to quantify harm - often how large is
potential harm
 63% fail to evaluate the quality of the evidence
 57% fail to compare new idea with existing options

9. Kid-in-candy-store picture
of U.S. health care
 70% of ideas are terrific
 70% are risk-free
 70% - cost not an issue

10. A clear pattern
 News stories tend to emphasize or
exaggerate the benefits of treatments,
tests, products and procedures.
 And they tend to minimize or ignore
the harms.

11. Other themes:
 Failure to convey limits of observational studies
 Rush to report on talks at scientific meetings
 Common flaws in health business stories
 Single source stories
 No independent perspectives
 Coverage of screening issues

12. Concerns about cumulative effect of health messages
 Daily drumbeat of news
 Steady diet of news from journals - disregard for
publication bias for positive findings
 Advertising
 Web health info
 News releases from industry, academic medical
centers, even from journals
 Misleading Twitter messages

13. Billboard
Bombardment

14. Prizes for Prostates

15. Bats & balls &
urinals

17. Actual newspaper headlines on
DaVinci Robotic Surgery
 Cancer survivors meet lifesaving surgical robot
 Da Vinci puts magical touch on the prostate
 Robotic surgeon's hands never tremble
 Da Vinci is code for faster recovery
 Hospital hopes robot surgery will lure patients
 WHAT’S THE HARM OF THIS?

18. Stories about wonderful health care
gifts are NOT what they appear to be.

19. Stories of robotic regret,
complications, costs starting to pile up
 Long learning curve for surgeons – one analysis showed that 3
experienced surgeons needed more than 1,600 cases to achieve
acceptable outcomes with robotic-assisted laparoscopic
prostatectomy
 With 2,000 of these $ million machines in place, how many
surgeons have done 1,600 prostates?
 A 2010 study showed robot-assisted hysterectomies were
associated with longer surgical times and cost an average of
$2,600 more.
 Bloomberg: A urologist confesses he was seduced by the robot.
But now rails against hospitals “tawdry marketing more familiar on
late-night infomercials by using patient testimonials. "I cannot believe
how quickly I recovered," a vigorous-looking patient is quoted as
saying. As a surgeon I have to ask: Where is the data?”

20. Story in Family Practice News 2 weeks ago…
 A recent explosion of interest in robotic surgery for
routine hysterectomies and treatment of other non-
oncologic gynecologic conditions is raising concern about
the cost and comparative value of the robot over
conventional laparoscopy.
 In 2010, for the first time, more hysterectomies were
performed with the da Vinci Surgical System than any
other procedure, including prostatectomy.
 There have not been any large-scale randomized trials of
robot-assisted surgery, and the "limited observational
evidence fails to show that the long-term outcomes of
robot-assisted surgery are superior to those of
conventional procedures."

21. The high-speed train of expensive,
unproven technologies has left the station
and it can’t be stopped.
 Robotic surgical systems
 Proton beam radiation facilities
 Intensity-modulated radiation therapy facilities

23. In print journalism in one week we saw.....
 Two stories that used drug company VPs as main
sources, one predicting a “breakthrough” without
giving any evidence!
 One that was based solely on a news release on mouse
research to tout a new approach to fight bone loss!
 One proclaimed that a "treatment may work" after it was
tested on 9 pieces of tissue outside the body!
 Another anointed a cancer gene therapy as a
"breakthrough" after it had been tested on just three
tumor samples!

24. Were these in a student newspaper?
 New York Times
 Los Angeles Times
 Reuters wire service
 HealthDay wire service

25. Costs

26. From BreastCancer.org discussion board

28. To prevent a single prostate cancer
(that may not even be a killer):
71 men
x 365 days/yr.
x 7 yrs.
x $3/pill =
$544,215
 And with prostate cancer, there’s a good chance it
wouldn’t have been a deadly cancer anyway!

29. Excellent job by Duff Wilson/NYT

30. Disease-mongering
MSNBC – Oct. 4, 2011 – Plastic surgeon wants to fix your 'runner's face’
Runners, beware. A New Jersey cosmetic surgeon has pinpointed
something more nefarious than shin splints, stress fractures and even dead
butt syndrome: "runner's face."
…that skinny, Skeletor-esque look some dedicated runners may unwittingly
develop, as they're likely distracted with all the miles they have to cover to
prepare for their next race. (The surgeon) describes the horrors thusly, in a
press release issued today:
Runner’s face generally occurs in both men and women ages 40+ who
exercise to improve their body, and in doing so end up with a skeletal and
bony face.
He suggests a Botox-Restylane (or other injectable filler) combo, which will
smooth wrinkles and plump that gaunt face right back up.

31. Which ―benefits‖ matter?
• Framing surrogate markers or intermediate endpoints
(test scores, blood values, etc.) as if they were the
outcomes that people should really care about to
establish improved quality of life or longevity.
• Perhaps the Avastin story is best example where
limitations of progression-free survival were often not
adequately explained.
“will not make you live longer, won't make you feel better, and
may have life-threatening side effects, but it will keep your cancer
from worsening by an average of 1 to 2 months."

32. Observational studies: "Women who drank >3 cups of coffee
per day had a 20% lower risk of developing basal cell
carcinoma than women who drank <1 cup/month"
 WebMD: ―Coffee Fights Common Skin Cancer‖
 ‖3 cups of coffee a day to help keep skin cancer away?" asks
MSNBC.
 ‖Coffee Lowers Risk of Skin Cancer" announced Medical News
Today.
 ‖Coffee cuts skin cancer risk,‖ stated the Washington Post
 The American Association for Cancer Research news release
said the only thing you really can say: ―Coffee consumption
associated with decreased risk for basal cell carcinoma‖
 If that's too boring, then don't report the story because
otherwise you're simply wrong.

33. Rush to report on talks at
scientific meetings
Last month’s American Heart Assoc. Scientific Sessions: experimental stem
cell intervention in 14 people - only 8 of whom were followed for a year.
 Tyranny of the anecdote: All 3 TV networks & WebMD rode the same
glowing patient anecdote
 Money quote keeps paying dividends: The same researcher quote -
"biggest advance in cardiovascular medicine in my lifetime" - appeared in
the ABC and NBC stories.
 EXCITEMENT!
 WebMD reported that one researcher they interviewed could barely contain his
excitement.
 ABC anchor Diane Sawyer discussed on the air how "excited" correspondent
Richard Besser was. Is that the job of journalism to convey how excited a
reporter is?
 Anchor Sawyer also gushed this question to Besser, "Is this the real prospect that
the nation's number one killer could be cured?"
 CBS used cure and breakthrough in the same headline.

34. Health business story
Single source
No independent perspective
Reuters: Medtronic tests stent for erectile dysfunction
October 20, 2011
Is ―an initial feasibility study in 30 men‖ worth reporting?
With a company-sponsored investigator as the only source?
―30 million US men and 300 million worldwide with erectile
dysfunction‖
• That counts ALL degrees of erectile dysfunction but not all = a total
inability to achieve an erection.
• Many different causes and vascular causes might not be safely
amenable to stenting.
• So, even in a business story, the "market" of "a real unmet clinical
need" for a surgical procedure may have been inflated in this story.

35. News coverage & poor public discussion of
screening issues should be most concerning.
Worst, most biased coverage I’ve seen in 37 years

36. 10 claims that many stories didn’t challenge
1. This is all about saving money.
2. This is about rationing.
3. It’s the first sign of Obamacare.
4. Government is deciding some lives don’t matter.
5. Early detection always saves lives.
6. The fact that I or someone I know was saved by a mammogram proves
that more testing is better.
7. The shifting recommendations prove that scientists are clueless.
8. This was surprising and came out of nowhere.
9. The task force members must be uninformed idiots - especially since
none was an oncologist or radiologist.
10. American Cancer Society disagrees so the task force must be wrong.

37. What the Task Force actually wrote:
“The decision to start regular, biennial screening
mammography before the age of 50 years
should be an individual one and take patient
context into account, including the patient's
values regarding specific benefits and harms.”

38. Annals of Internal Medicine editors
called it a “media cacophony”
 “Confusion, politics, conflicted
experts, anecdote, and emotion ruled
front pages, airwaves, the Internet,
and dinner-table conversations.”

39. The Public's Response to the U.S. Preventive
Services Task Force's 2009 Recommendations
on Mammography Screening
―These results are consistent with previous
studies that suggest a media bias in favor of
mammography screening."

40. 37-year Chicago Tribune & NY
Times veteran John Crewdson wrote:
 “There are multiple reasons women are ill-
informed about breast cancer. The fault lies
primarily with their physicians, the cancer
establishment, and the news media--especially the
news media. Until coverage of breast cancer rises
above the level of scary warnings mixed with
heartwarming stories of cancer survivors, women
are likely to go on being perplexed."

41. A national poll suggests lingering
impact of imbalanced news
• 11 percent of women surveyed said they
believe mammograms should start in the
20s, even for women with no risk factors
• 29 percent believe mammograms should
start in their 30s.
• 45 percent said the guidelines were an
attempt to reduce health-care costs

42. Other examples:
 Chicago Sun-Times
 Wall Street Journal
 Washington Post
 New York Daily News
 Minneapolis Star Tribune
 All TV networks

43. The ink
isn’t even
dry on
studies
before
marketing
begins

44. On Twitter
 Men’s Health magazine:
If you're a smoker, you NEED to get a CT scan.
Body of story: ―But don't run out and ask for one.‖ (as it
discusses issues of costs and false positive test results)
 American Association of Retired Persons:
Are you a smoker? CT scan those lungs
Body of story: 371 words of caveats.

45. HealthDay wire service didn’t challenge
researcher promoting universal pancreatic
cancer screening for everyone over 50.
 After study of tissue from 7 people!

46. Journalists’ Screening Bias
 Crusading
 one-sided
 advocacy

47. Why don’t we deliver this message?
“All screening tests
cause harm;
some may do good.”
But much health journalism consistently emphasizes
benefits & minimizes harms

49. And that’s the message I
think we’re missing
 That there’s a harm behind screening promotions that
are not based on evidence and that do not fully
inform readers/consumers/would-be patients
 That there’s a harm in selling sickness…in selling the
search for weapons of mass destruction inside all of
us
 Communication of the evidence about tradeoffs –
about harms - is a key health policy issue

50. 2007 Cochrane Review of
randomized clinical trials of
screened versus not screened –
only small benefits from screening
mammography

51. Breast Self Exam
 2 randomized controlled trials have
shown no benefit from breast self exam
in decreasing mortality
 Breast self exam does lead to the
discovery of more benign masses

52. Screening - Myth and Truth
 Myth: if women go for their recommended
screening, their breast cancer will be caught early
when it is 98% curable
 Truth:
 Many breast cancers are not picked up on screening
 Screening decreases the risk of dying of breast cancer
by only 15-30%
 Screening best finds the slower growing less aggressive
cancers with better prognosis
 Faster growing, more aggressive tumors (biology!) are
more likely to be found between screens and higher
numbers have metastasized prior to diagnosis

53. What pinkwashing leaves out

54. The National Breast Cancer Coalition talks
about ―changing the conversation‖
 Measure real progress, not by counting how
many women have been screened or what the
five-year survival rates are but by seeing
significant reductions in incidence and
mortality rates.

55. With screening messages,
we see a clash between…
 Science  Intuition
 Evidence  Emotion
 Data  Anecdote
 Recommendations for  Decision-making by an
entire population individual
 Grasping uncertainty  Promoting false certainty
 What we can prove  What we believe, wish, or
hope

56. ―I honestly believe it is better to know nothing
than to know what ain’t so.‖
Josh Billings (pen name of humorist
Henry Wheeler Shaw, 1818 – 1885)

57. Pitfalls of a steady diet of
journal stories
PLoS Med 2005; 2(8): e124

58. The problem begins with the public’s rising expectations of
science. Being human, scientists are tempted to show that
they know more than they do. The number of investigators—
and the number of experiments, observations and analyses
they produce—has also increased exponentially in many
fields, but adequate safeguards against bias are lacking.
Research is fragmented, competition is fierce and emphasis is
often given to single studies instead of the big picture. Much
research is conducted for reasons other than the pursuit of
truth. Conflicts of interest abound, and they influence
outcomes.

60. Spinning results of
randomized clinical trials
Boultron et al, JAMA May 26, 2010, 303 (20): 2058
In this representative sample of RCTs published in
2006 with statistically non-significant primary
outcomes, the reporting and interpretation of findings
was frequently inconsistent with the results.

61. Reporting bias in medical research
- a narrative review
 McGauran et al. Trials 2010, 11:37
 We identified reporting bias in 40 indications
comprising around 50 different
pharmacological, diagnostic, and preventive
interventions. Many cases involved the withholding
of study data by manufacturers and regulatory
agencies or the active attempt by manufacturers to
suppress publication. The ascertained effects of
reporting bias included the overestimation of
efficacy and the underestimation of safety risks of
interventions.

62. Reason for optimism
 New models such as ProPublica, Kaiser Health News
 Smart bloggers – including
physicians, scientists, hospital administrator
 Training
 This boot camp
 NIH Medicine in the Media workshops
 Association of Health Care Journalists
 USC Annenberg California Endowment fellowships

63. Regular columns analyzing evidence
 LA Times’ monthly ―Healthy Skeptic‖ column
(Seeing Through Skin Supplements’ Claims)
 New York Times ―Really?‖ weekly column
(The Claim: Chewing Gum Can Prevent Ear Infections)
 WSJ ―Research Report‖ biweekly column – 6 studies
each time
 Washington Post ―Quick Study‖ weekly column
(Vibrating machines seem to offer no benefit for older
women’s bones )

64. Thank you
gary@healthnewsreview.org