NCE Symposium, June 27, 2016. Eddy Nason gives an overview of why impact should be measured, the NCE's and impact, frameworks for assessing impact, and impact assessment tools.

1. SCIENCE
UNDER
THE
MICROSCOPE
MONITORING
AND
EVALUATING
IMPACT

2. Why
measure
impact?
Not
just
because
you’re
told
to…
June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 2

3. Impact
Assessment
Purpose
June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 3
Analysis
Advocacy
Accountability
Allocation

4. Purpose…
ADVOCACY
MAKING
THE
CASE
FOR
RESEARCH
ACCOUNTABILITY
TO
TAXPAYERS
&
DONORS
June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 4
“Used
as
evidence
as
part
of
the
preparation
for
the
spending
review
and
will
be
in
the
foreseeable
future” –
Science
Minister
“few
studies
that
have
made
a
genuine
attempt
to
objectively
assess
the
economic
returns
of
research” –Nature
Editorial

5. Purpose…
ANALYSIS
UNDERSTANDING
WHAT
WORKS
ALLOCATION
REWARDING
IMPACT
June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 5
Researchers
that
show
interests
in
other
field(s)
have
a
higher
impact
and
those
focused
on
a
single
topic
have
a
lower
wider
social
impact

6. There
are
some
perennial
challenges
to
research
evaluation
June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 6

7. NCEs
and
Impact
What’s
currently
happening
and
why?
June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 7

8. What
is
an
impact
for
NCEs?
Potential
contentious
issue:
• In
a
number
of
research
impact
models,
impacts
are
often
‘things
that
arise
as
a
consequence
of
the
research’
(i.e.
outputs
and
outcomes)
• For
more
traditional
evaluation,
impacts
are
often
‘long-­‐term
outcomes’
(i.e.
societal
/
behaviour
changes
as
a
result
of
the
research)
• For
NCEs,
the
publication
of
impacts
in
annual
reports
seems
to
suggest
the
former
definition
• The
‘What
if’
report
catalogues
numbers
of
publications,
partnerships,
co-­‐
funding
levels,
capacity
building
opportunities,
patents.
• There
is
an
argument
that
NCEs
should
be
looking
at
impacts
as
more
the
latter
definition
• The
stated
goal
of
NCEs
is
to
improve
the
economy
and
quality
of
life
for
Canadians
June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 8

9. How
to
know
if
you’re
having
an
impact?
Two
questions
really:
◦ 1
– How
would
I
measure
/
collate
the
impacts
◦ 2
– How
can
I
find
out
about
the
impacts?
These
two
questions
have
different
answers
◦ 1
– Using
frameworks,
methods
and
approaches
around
impact
assessment
◦ 2
– By
engaging
with
the
right
people
to
know
‘of’
the
impacts
that
occur
June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 9

10. Frameworks
Creating
a
structure
to
assess
research
impact…
June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 10

11. Canadian
Academy
of
Health
Sciences
June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 11

12. The
Co-­‐produced
Pathway
to
Impact
Based
on
a
knowledge
mobilization
approach
◦ Combining:
university
push,
community
pull,
knowledge
exchange,
and
co-­‐
production
of
knowledge
Logic
model
driven
◦ Research
à dissemination
à uptake
à implementation
à impact
Developed
with
PREVNet to
take
advantage
of
their
co-­‐creation
model
for
research
(academic
and
community
input)
At
each
stage
of
the
logic
model
can
identify
‘benefits’
of
PREVNet
activities:
June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 12
Research
•Research
benefits
Dissemination
•Dissemination
benefits
Uptake
•Uptake
benefits
Implementation
•Implementation
benefits
Impact
•‘Traditional’
outcomes

13. Research
Excellence
Framework
(REF
-­‐ UK)
The
primary
purpose
of
the
REF
is
to
produce
assessment
outcomes
for
institutions
◦ To
inform
the
selective
allocation
of
research
funding
to
HEIs,
with
effect
from
2015-­‐16
◦ To
provide
accountability
for
public
investment
in
research
and
produces
evidence
of
the
benefits
of
this
investment
◦ To
provide
benchmarking
information
Research
Excellence
Framework
(REF)
collected
impact
case
studies
from
university
research
Universities
asked
to
provide
narratives
of
impacts
Based
on
expert
review
of
institution
submissions
20%
of
score
is
based
on
‘Impact’
◦ Criteria
for
assessing
impacts
are
‘reach’
and
‘significance’
June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 13

14. Comparing
frameworks
CAHS Co-­‐produced REF
Strengths Comprehensive,
Process and
outputs
and
impacts,
Indicator
library
KM
framework
base,
Strong narrative
of
impacts
Rich
detail,
Across
disciplines,
Ability
to
compare
impact
stories
Weaknesses Resource
intensive,
Requires
expertise
to
implement,
Reliance
on
user
input,
difficult
to
identify
‘unit’
of
impact
Not
representative,
highly
resource
intensive
to
collect
Opportunities International
comparisons,
flexibility,
Ability to
describe
the
path
to
impacts,
Building
on
growing
KM
field
Developing library
of
impact
narratives,
analysis
opportunities
Threats No
implementing
owner
Could be
seen
as
relevant
only
to
co-­‐
production
research
Linking
impact
to
funding
politically
challenging
June
27
2016 FOOTER
TEXT
HERE 14

15. Who
knows
the
impacts?
Identifying
the
right
people
to
get
at
your
impacts…
June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 15

16. Sphere
of
influence
and
knowledge
Beneficiary
User
Researcher
Input
• Research
funding
Process
• Research
activity
Output
• Research
product
Outcome
• Changed
decisions
June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 16
ControlInfluence

17. Tools
to
assess
impact
Can
anyone
make
this
easier
for
you?
June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 17

18. Some
examples
of
impact
tools
Research
fish
ImpactFinder
ImpactStory
Uber
Research
June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 18

19. Articulating
impact
How
you
say
it
can
be
more
important
than
what
you
say…
June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 19

20. REF
examples
June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 20
0 1,000+
624
Ireland
4
Bermuda
1
Caicos
Islands
25
Jamaica
24
Macedonia
(FYROM)
15
Moldova
209
Russian Federation
1,822
United States
878
Canada
10
Greenland
48
Iceland
159
Mexico
5,308
United
Kingdom
361
Switzerland
281
Denmark
141
Portugal
175
Finland
28
Latvia
52
Estonia
44
Lithuania
8
Gibraltar
37
Morocco
29
Kazakhstan
50
Croatia
59
Slovenia
18
Mongolia
619
China
7
North Korea
397
Sweden
321
Norway
19
Liberia
84
Ghana
2
Western
Sahara
24
Sierra Leone
3
Guinea-Bissau
21
Tunisia
38
Malta 27
Lebanon
18
Kyrgyzstan
18
Iran
38
Iraq
54
Qatar59
Saudi
Arabia
5
Turkmenistan
6
Bhutan
46
Nepal
20
Algeria
23
Libya
72
Egypt
India
492
58
Ethiopia
Pakistan
72
Afghanistan
55
43
Sri Lanka
3
Cambodia
8
Laos
44
Vietnam
86
UAE
24
Kuwait
50
Jordan
2
British Indian
Ocean Territory
1,078
Australia
358
New Zealand
1
New
Caledonia
3
Fiji
10
Niger
7
Chad
26
Sudan
14
South
Sudan
15
Mali
24
DR Congo
4
Central African
Republic
262
South
Africa
8
Swaziland
10
Madagascar
15
Mauritius
32
Zimbabwe
28
Mozambique
1
Comoros
4
Djibouti
35
Zambia
21
Namibia
26
Bolivia
56
Colombia
80
Argentina
39
Peru
28
Ecuador
81
Chile
4
Paraguay
15
Venezuela
4
Saint
Lucia
9
Haiti
6
Dominican
Republic
5
Puerto Rico
1
British Virgin Islands
2
U.S. Virgin Islands
3
Anguilla
10
Barbados
8
Trinidad and
Tobago
1
Grenada
10
El Salvador
6
Belize
9
Guatemala
14
Papua
New Guinea
56
Philippines
4
Tonga
1
American
Samoa
11
Brunei
1
Palau
1
Guam
1
Cook Islands
182
Poland
86
Taiwan
678
France
864
Germany
603
Netherlands
484
Italy
56
Romania
52
Bulgaria
458
Spain
1
Andorra
443
Japan
393
Belgium
26
Luxembourg
320
Brazil
216
Austria
165
Turkey28
Syria
63
Ukraine
161
Israel
40
Palestinian
Territory
155
Singapore
136
Greece
128
Hong Kong
4
Macao
126
Malaysia
108
Kenya
106
Czech Republic
38
Slovakia
105
South Korea
100
Indonesia
96
Thailand
95
Hungary
41
Serbia
77
Uganda
81
Nigeria
80
Tanzania
45
Malawi
72
Bangladesh
1
Cayman Islands
22
Senegal
11
Gambia
9
Maldives
28
Rwanda
8
Burundi
22
Bosnia &
Herzegovina
13
Montenegro
20
Kosovo
21
Albania
22
Myanmar
21
Somalia
20
Botswana
18
Azerbaijan
7
Georgia
19
Armenia
19
Cameroon
2
Equatorial
Guinea
9
Gabon
5
Congo –
Brazzaville
18
Oman
16
Burkina Faso
16
Uruguay
15
Cuba
14
Uzbekistan
13
Bahrain
12
Yemen
11
Panama
10
Belarus
10
Guernsey
9
Guatemala
8
Ivory Coast
4
Togo
6
Benin
8
Seychelles
10
Vanuatu
8
Solomon
Islands
7
Angola
7
Tajikistan
7
Guyana
1
French
Guyana
1
Surinam
6
Lesotho
6
Monaco
6
Montserrat
5
Dominica
4
Guinea
3
Cape
Verde
3
East
Timor
3
Eritrea
3
Falkland
Islands
2
Mauritania
1
Bahamas
1
Saipan
53
Cyprus
15
Nicaragua
2
Honduras
http://impact.ref.ac.
uk/CaseStudies/

21. REF
examples
June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 21
UoA
23
Uo
A
5
UoA 1UoA 36
UoA 35
UoA 34
UoA
33
UoA
32
UoA
31
UoA30
UoA29
UoA28UoA27
UoA26
UoA25
UoA
24
UoA
22
UoA21
UoA20
UoA19 UoA18
UoA17
UoA16
UoA
15
UoA
13
UoA12
UoA11
UoA10UoA9
UoA8
UoA7
UoA6
UoA
4
UoA 3
UoA 2
UoA
14
0
%
4
%
42%0%
0%
0%
0%
1
%
0%
0%
0%
0%0%
6%
0%
0%
0%
0%
0%
0% 1%
0%
0%
2%
0%
0%
1%
3%0%
1%
0%
2%
11%
14%
29%
0
%
0%
Clinical Guidelines
10%
20%
30%
40%
50%
60%
70%
80%
100%
90%
10%
20%
30%
40%
50%
60%
70%
80%
100%
90%
10%
20%
30%
40%
50%
60%
70%
80%
100%
90%
10%
20%
30%
40%
50%
60%
70%
80%
100%
90%
‘Clinical guidance’
(guidelin patient clinic treatment recommend
stroke nice risk trial)
n = 326 or c5% of case studies
‘Informing government policy’
(develop polici nation plan govern inform
work strategi assess)
(n=1233 or c20% of case studies)
UoA
23
Uo
A
5
UoA 1UoA 36
UoA 35
UoA 34
UoA
33
UoA
32
UoA
31
UoA30
UoA29
UoA28UoA27
UoA26
UoA25
UoA
24
UoA
22
UoA21
UoA20
UoA19 UoA18
UoA17
UoA16
UoA
15
UoA
13
UoA12
UoA11
UoA10UoA9
UoA8
UoA7
UoA6
UoA
4
UoA 3
UoA 2
UoA
14
31
%
15
%
8%15%
4%
6%
1%
5
%
3%
3%
1%
8%3%
25%
42%
30%
41%
18%
24%
36% 15%
43%
51%
7%
3%
6%
9%
11%1%
4%
35%
26%
23%
25%
36%
33
%
Informing Government Strategy
0%
10%
20%
30%
40%
50%
60%
70%
80%
100%
90%
10%
20%
30%
40%
50%
60%
70%
80%
100%
90%
10%
20%
30%
40%
50%
60%
70%
80%
100%
90%
10%
20%
30%
40%
50%
60%
70%
80%
100%
90%

22. June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 22
REF
examples

23. June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 23
REF
examples

24. REF
examples
June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 24

25. Impact
Array
June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 25

26. The
journey
is
still
going
on…
Eddy
Nason,
Assistant
Director,
Ontario
SPOR
Support
Unit
E:
eddynason@ossu.ca @eddynason
June
27
2016 SCIENCE
UNDER
THE
MICROSCOPE 26