Managing Risk and Ensuring Effective Oversight in Outsourced Clinical Trials

1. Managing Risk and
Ensuring Effective
Oversight in Outsourced
Clinical Trials
Clinical Quality
Oversight Forum

2. 2
Panel Members
Mitch Katz
Executive Director Medical Research Operations, PURDUE PHARMA LP
David Marks
Executive Vice President, Quality Management, RESEARCH PHARMACEUTICAL
SERVICES, INC. (RPS)
David Nickerson
Senior Director, Clinical Quality Management, PFIZER
Clinical Quality Oversight Forum

3. 3
Today’s Topics
• Background: Challenges in quality management
• Part One: Proactive quality management and
effective oversight
• Part Two: Risk sharing
• Part Three: Risk assessment and risk management
Clinical Quality Oversight Forum

4. Background and
Overview

5. 5
Mission: To accelerate the development of best practice
approaches and industry standards for the proactive quality
management of outsourced trials
Lead Sponsors and CROs in optimizing their approaches to proactive quality
management with an emphasis on bringing them into greater alignment.
Avoca Quality Consortium

6. 6
Avoca Quality Consortium
Pharma/Biotech Industry Participation to Date
**Roche is also Consortium Members

7. 7
Avoca Quality Consortium
CRO Industry Participation

8. 8
Are there quality issues
with outsourced
clinical trials?
Challenges in Quality Management

9. 9
Challenges in Quality Management
Disconnect between sponsors and CROs regarding perceptions of
quality delivered by CROs
Sponsor: Overall, how satisfied are you
with the quality delivered by your
Clinical Service Providers (including but
not limited to CROs)?
2%
45%
30%
20%
3%
17%
70%
10%
3%
CRO: Overall, how satisfied are you
with the quality that your company
delivers for its sponsors?
N=245* N=200*
5%
13%
8%
8%
15%
15%
40%
10%
3%
11%
27%
95%
100%
67%
77%
83%
64%
58%
20%
60%
71%
57%
66%
60%
20%
15%
15%
20%
20%
20%
14%
23%
16%
10%
8%
6%
7%
20%
10%
14%
17%
7%
2%
0% 20% 40% 60% 80% 100%
A
B
C
D
E
F
G
H
I
J
K
-All-
Very satisfied Somewhat satisfied Neither satisfied nor dissatisfied
Somewhat dissatisfied Very dissatisfied
*2013 Avoca Quality Consortium Assessment

10. 10
Overview: Challenges in Quality Management
“Reactive” vs “proactive” approaches to quality management: a
look at whether quality issues exist
Areas with relatively high satisfaction:
● Compliance with SOPs and other written procedures
● Data quality and integrity
● Audit plans and execution
Areas with relatively high dissatisfaction:
● Oversight of third party vendors
● Governance of quality
● Communications surrounding quality
● Availability of quality personnel for projects
● Efficiency/timeliness in achieving clean data
● Adherence to monitoring plan
3.3 – 3.5
(on scale of 1 to 5)
2.9 – 3.2
(on scale of 1 to 5)

11. 11
Quality by Task: Comparison of Sponsor and CRO Results
(2012 Avoca Quality Consortium Assessment)
Sponsors:
On average, how satisfied are you with
your clinical service providers’…
CROs:
On average, how would you rate your
company's…
Compliance with SOPs and other written
procedures
3.5 3.7
Data quality and integrity 3.4 3.8
Audit plans and execution 3.3 3.6
Adequacy of Monitoring Plan 3.2 3.5
Document control 3.2 3.3
Adherence to Monitoring Plan 3.2 3.4
Efficiency/timeliness in achieving clean data 3.2 3.5
Monitoring of protocol compliance 3.1 3.7
Availability of quality personnel for my projects 3.1 3.2
Management of protocol compliance 3.1 3.6
Staff training 3.0 3.3
Site training 3.0 3.4
Governance of quality (e.g. accountability,
management system, leadership support)
2.9 3.4
Oversight of third party vendors (e.g. labs, IVRS
vendors, etc.)
2.9 3.3
Communications surrounding quality 2.9 3.4

12. 12
Research Finding:
Strong correlation
between taking
proactive approaches
to quality management
and satisfaction with
the quality of
deliverables.
Sponsor Proactivity Drives Satisfaction
12
0
1
2
3
4
5
0
1
2
3
4
5
Sa*sfac*on
with
the
"quality"
delivered
by
clinical
service
providers
The
(sponsor)
teams
with
which
I
work
iden*fy
all
processes/deliverables
for
which
quality
expecta*ons
of
clinical
service
providers
need
to
be
set
0
1
2
3
4
5
0 1 2 3 4 5
Satisfactionwiththe"quality"
deliveredbyclinicalservice
providers
The (sponsor) teams with which I work effectively communicate
their expectations regarding quality to their
clinical service providers

13. 13
Clarity of expectations: scores
range between 2.4 on the low end to
4.8 on the high end (based on a
scale of 1 to 5)
Avoca Relationship Assessments
Clarity of Roles and Expectations vs.
Sponsor’s Overall Satisfaction:
Across Strategic Partnerships*
*p<.05
Statistically significant
correlation across strategic
partnerships:
Clarity of expectations, roles,
and responsibilities, and
overall satisfaction with the
work performed.

14. 14
Data suggests room for improvement in
proactive approaches
Many sponsors companies have reported
difficulty in operationalizing approaches for
identifying and communicating expectations.
The (sponsor) teams with which I work effectively communicate their
expectations regarding quality to their clinical service providers.

15. 15
Operationalizing Proactive Communication

16. 16
Operationalizing Proactive Communication

17. 17
Operationalizing Proactive Communication
Construction Industry
ü Has operationalized
proactive, systematic and
integrated communication
strategies
Although buildings are more
complex and sophisticated,
building failure is exceedingly rare:
(.00002 percent/year)
“The biggest cause of serious
failure in this business is a failure
of communication.”
Finbarr O’Sullivan, Project Executive, Moriarty
& Associates, Project Executive, Russia Wharf office
building on Boston Waterfront

18. 18
Leuchten’s Art Studio/Garage
Subset of problems to
date:
ü Foundation footings
poured based on old set
of drawings
ü Scheduling snafu’s
between builder and
roofer
ü Failed inspection due to
issues with electrical and
plumbing
Root causes of issues:
Problems
with proactive
communication,
collaboration and clarity
of expectations

19. 19
Themes to explore today
• Proactive communication
• Collaboration
• Clarity of expectations
...and impact on quality of outsourced
clinical trials
Clinical Quality Oversight Forum

20. Part One:
Proactive quality
management and
effective oversight

21. 21
• Research Topics
– Micromanagement
– Documentation around roles and responsibilities
– Overseeing work at a strategic level rather than tactical
– Definition of roles to minimize duplication of effort
– Efficiency in use of resources
– Use of technology to promote efficient oversight practices
Proactive Quality Management and
Effective Oversight
`

22. 22
What do you think?
Polling question for audience:
Sponsors: My company is efficient in the use of resources applied to
outsourced projects.
CROs: Sponsors that I work with are efficient in the use of resources applied
to outsourced projects.
A. Strongly agree
B. Agree
C. Neither agree nor disagree
D. Disagree
E. Strongly disagree

23. 23
21%
13%
5%
6%
6%
2%
30%
42%
34%
34%
28%
18%
34%
21%
34%
34%
29%
31%
11%
20%
18%
18%
32%
34%
4%
5%
8%
8%
6%
15%
The operational teams involved in overseeing our CROs
consistently micromanage to ensure that the quality of
deliverables meets expectations.
My company clearly documents CRO oversight practices,
roles, and responsibilities.
I personally have received high-quality deliverables from
CROs, even when overseeing their work at a "strategic"
rather than detailed level.
My company clearly defines the roles of internal and CRO
staff so as to minimize duplication of effort.
My company is efficient in the use of resources applied to
the oversight of outsourced trials.
The technology systems used by my company and its CRO
partners promote efficient oversight practices.
Sponsor Perceptions of Oversight Practices
Mean N
3.5 219
3.4 219
3.1 203
3.1 218
2.9 218
2.6 215
Perceptions of Oversight Practices
Consortium Sponsor Findings
5
Strongly Agree
4 3
Neither Agree
nor Disagree
2 1
Strongly Disagree

24. 24
8%
5%
3%
4%
3%
4%
47%
34%
30%
14%
23%
22%
32%
34%
33%
44%
35%
29%
11%
23%
25%
32%
31%
29%
3%
5%
10%
5%
8%
16%
The sponsors with which I work consistently micromanage
to ensure that the quality of deliverables meets
expectations.
The sponsors with which I work clearly document CRO
oversight practices, roles, and responsibilities.
The technology systems used by my company and its
sponsor partners promote efficient oversight practices.
The sponsors with which I work are efficient in the use of
resources applied to the oversight of outsourced trials.
The sponsors with which I work clearly define the roles of
internal and CRO staff so as to minimize duplication of
effort.
There is a direct positive relationship between the
intensity of sponsor oversight and the quality of CRO
deliverables.
CRO Perceptions of Sponsor Oversight Practices
Mean N
3.5 157
3.1 153
2.9 159
2.8 152
2.8 149
2.7 158
Perceptions of Oversight Practices
CRO Findings
5
Strongly Agree
4 3
Neither Agree
nor Disagree
2 1
Strongly Disagree

25. 25
Statements Evaluated by Sponsors
Sponsor
Average
CRO
Average
Statements Evaluated by CROs
The operational teams involved in overseeing our CROs
consistently micromanage to ensure that the quality of
deliverables meets expectations.
3.5 3.5
The sponsors with which I work consistently micromanage
to ensure that the quality of deliverables meets
expectations.
My company clearly documents CRO oversight practices,
roles, and responsibilities.
3.4 3.1
The sponsors with which I work clearly document CRO
oversight practices, roles, and responsibilities.
My company clearly defines the roles of internal and CRO
staff so as to minimize duplication of effort.
3.1 2.8
The sponsors with which I work clearly define the roles of
internal and CRO staff to minimize duplication of effort.
I personally have received high-quality deliverables from
CROs, even when overseeing their work at a "strategic"
rather than detailed level.
3.1 2.7
There is a direct positive relationship between the
intensity of sponsor oversight and the quality of CRO
deliverables.
My company is efficient in the use of resources applied to
the oversight of outsourced trials.
2.9 2.8
The sponsors with which I work are efficient in the use of
resources applied to the oversight of outsourced trials.
The technology systems used by my company and its CRO
partners promote efficient oversight practices.
2.6 2.9
The technology systems used by my company and its
sponsor partners promote efficient oversight practices.
N≥203 N≥149
Perceptions of Oversight Practices
5 = Strongly Agree, 1 = Strongly Disagree
Perceptions of Oversight Practices
Sponsor and CRO Findings

26. 26
Best Practices in Quality Management
Operationalizing
approaches to
setting and
communicating
expectations
drives consistency and
increases quality

27. 27
Operationalizing Proactive Approaches to Quality
Management and Oversight
Governance
/
Organiza*onal
Construct
Technical
/
Project
Oversight
Processes
Communica*on
Oversight
Leadership
Requirements
Metrics
/
Analy*cs
/
Technology
Roles
/
Responsibili*es
Proac*ve
Risk
Management
/
Con*ngencies
Effective
Oversight
Define
Plan
Lead
Decide
Measure
Deliver
Proactive
Quality
Management

28. 28
Verbatim Comments
“[Sponsors don’t give us] ownership of decisions that we should have. This
delays our deliverables which Sponsors will hold us accountable for in the
end.” [CRO]
“CRO's don't like to problem solve and offer suggestions. They'd rather just
do as they are told.” [Sponsor]
“One Sponsor that I work with is new to fully outsource trials, so at the
beginning of our working relationship, they micromanaged us. This has
slowly relaxed as we showed that we were consistently meeting project
deliverables. “ [CRO]

29. Panel Discussion

30. Part Two:
Risk-sharing

31. 31
Risk-sharing between Sponsors and
Providers
Types of Risk-sharing
● Provider bonuses for achieving milestones and targets
● Provider penalties for missing milestones and targets
● Guarantees of continued work/revenue stream in
exchange for provider commitments
● Provider stake in outcome of study
(e.g., company stock)

32. 32
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%
Provider penalties for failure to achieve milestones/targets
Provider bonuses for achievement of milestones/targets
Guarantee of continued work/revenue stream in exchange for
provider commitments
None of the above
Strategic partnerships/alliances (N=205) Preferred provider relationships (N=194) Transactional relationships (N=217)
Risk-Sharing Models Used by Type of Relationship
Risk-Sharing Model Usage
Sponsor Findings
Risk-sharing models are used more commonly in strategic partnerships or
preferred provider agreements than in transactional relationships.

33. 33
What is your experience?
Polling question for audience:
With which risk-sharing method have you had the most positive
experience?
A. Provider bonuses for achieving milestones and targets
B. Provider penalties for missing milestones and targets
C. Guarantees of continued work/revenue stream in exchange for provider
commitments
D. Provider stake in outcome of study (e.g., company stock)

34. 34
41%
30%
25%
50%
62%
21%
48%
61%
52%
47%
34%
55%
10%
9%
23%
3%
4%
24%
Provider bonuses for achievement of milestones/targets
Guarantee of continued work/revenue stream in exchange
for provider commitments
Provider penalties for failure to achieve milestones/targets
Provider bonuses for achievement of milestones/targets
Guarantee of continued work/revenue stream in exchange
for provider commitments
Provider penalties for failure to achieve milestones/targets
Experience/Satisfaction with Risk-Sharing Models Used
Primarily
Positive
A mix of Positive
and Negative
Primarily
Negative
Risk-Sharing Satisfaction
Sponsor and CRO Findings
N
87
94
104
92
74
95
SponsorsCROs
Consor*um
sponsors
report
more
posi*ve
experiences
with
the
use
of
provider
bonuses
compared
to
provider
penal*es.

35. 35
41%
30%
25%
50%
62%
21%
48%
61%
52%
47%
34%
55%
10%
9%
23%
3%
4%
24%
Provider bonuses for achievement of milestones/targets
Guarantee of continued work/revenue stream in exchange
for provider commitments
Provider penalties for failure to achieve milestones/targets
Provider bonuses for achievement of milestones/targets
Guarantee of continued work/revenue stream in exchange
for provider commitments
Provider penalties for failure to achieve milestones/targets
Experience/Satisfaction with Risk-Sharing Models Used
Primarily
Positive
A mix of Positive
and Negative
Primarily
Negative
Risk-Sharing Satisfaction
Sponsor and CRO Findings
N
87
94
104
92
74
95
SponsorsCROs
Similar results are reported by sponsors and CROs in regard to use of provider
penalties for failure to achieve milestones/targets.

36. Panel Discussion

37. Part Three:
Risk Assessment and
Risk Management

38. 38
37%
18%
20%
13%
12%
Usage Frequency of Systematic Risk Assessment Processes
Systematic Risk Assessment Frequency
Sponsor and CRO Findings
51-75% of trials
>75% of trials
25-50% of trials
1-24% of trials
Never
52%
21%
13%
13% 1%
Sponsors CROs
N=174 N=135

39. 39
Use of Systematic Risk Assessment
“Please briefly describe the conditions under which a systematic risk
assessment process is used.” – Verbatim Responses
● “Not consistent”
● “Don’t know”
● “Often done ‘in silos’”
● Only…
╸ During RFP process
╸ At governance level
╸ Ad hoc
╸ At program level, not study level
╸ For audit strategy
╸ Upon sponsor request
╸ For key clients
╸ When resource limitations are a big concern
● Done, but not necessarily
“systematic and rigorous”
╸ Too high level and
“standard”
╸ copy and paste from
previous studies
╸ no time to think about
study-specific nuances

40. 40
Risk Assessment and Risk Management
“We do not consistently approach the work
from a risk assessment perspective and we
continually put ourselves in the position of
being surprised by sponsor questions,
demands or comments.” [CRO]

41. 41
Insights from Avoca Data
5 = Very Satisfied, 1 = Very Dissatisfied CONSORTIUM GROUP AVERAGES
Performance Aspect
In-house
Teams
CRO
Partners
CRO Self
Assessment
Proactive identification of potential risks 3.5 2.8 3.5
Proactive risk analysis and evaluation 3.3 2.7 3.4
Compilation of risk-related trial information
during a trial (observations, trends, etc.)
3.2 2.6 3.2
Frequency of review of risk-related trial
information
3.3 2.7 3.3
Rigor of review of risk-related trial
information
3.1 2.5 3.1
Communications regarding risk-related trial
information
3.3 2.8 3.2
Appropriateness of measures suggested or
taken in reaction to risk-related information
3.4 3.0 3.3
Overall performance on risk assessment and
management related activities
3.2 2.7 3.3
Low to middling
scores suggest
knowledge gaps in
key areas related to
risk identification,
analysis and
management
Performance Satisfaction on Select Risk Assessment Variables

42. 42
Systematic Risk Assessment Results
Sponsor and CRO Findings
More efficient use of resources for your company and/or your partner?
Increased quality?
Q: In general, have your risk assessment and management approaches resulted in…
N
204
159
203
159
9%
16%
20%
28%
39%
40%
36%
43%
17%
16%
8%
6%
35%
28%
36%
23%
Sponsors
CROs
Sponsors
CROs
Yes Sometimes No Don’t Know /
Too Soon to Tell

43. 43
Systematic Risk Assessment Results
Sponsor and CRO Findings
More efficient use of resources for your company and/or your partner?
Increased quality?
Q: In general, have your risk assessment and management approaches resulted in…
N
204
159
203
159
9%
16%
20%
28%
39%
40%
36%
43%
17%
16%
8%
6%
35%
28%
36%
23%
Sponsors
CROs
Sponsors
CROs
Yes Sometimes No Don’t Know /
Too Soon to Tell

44. 44
What is your experience?
Polling question for audience:
How would you rate your understanding of best practices in risk
assessment and management in clinical trials?
A. Very strong understanding
B. Good understanding
C. Fair understanding
D. Poor understanding

45. 45
Understanding of Best Practices in Risk Assessment/
Management
8%
39%
42%
11% Very strong
understanding
Good
understanding
Fair
understanding
Poor
understanding
How would you rate your understanding of best practices in risk
assessment and management in clinical trials?
9%
49%
35%
7% Very strong
understanding
Good
understanding
Fair
understanding
Poor
understanding
N=207* N=159*
Sponsor: CRO:
*2013 Avoca Quality Consortium Assessment

46. 46
“Team members inability to understand the big picture of what their
contribution means to the broader study and what the broader study
means to the overall strategy for the product is a big contributor to
failed risk management.” [Sponsor]
“We don't proactively manage risk effectively. Instead we
micromanage our CROs in the name of sponsor oversight, which is not
a good use of anyone's time or resources.” [Sponsor]
Verbatim Comments

47. Panel Discussion

48. Thank you!
Patricia Leuchten
patty.leuchten@theavocagroup.com
Mitch Katz
Mitchell.Katz@pharma.com
David Marks
DMarks@RPSweb.com
Dave Nickerson
David.F.Nickerson@pfizer.com