Blue Marble Evaluation โดย Michael Quinn Patton ในโครงการพัฒนากลุ่มความร่วมมือ (consortium) นักประเมินการวิจัยและนวัตกรรมแห่งประเทศไทย จัดโดย สถาบันนโยบายเศรษฐกิจการคลัง ร่วมกับ สกสว.

2. Blue Marble Images

4. 4
1978
2020

6. Think globally,
Act locally & globally,
Evaluate the
Interactions

7. EvalPartners
The international
movement to
strengthen national
evaluation capacities

8. MDGs 2000-2015

10. Compare images

12. Global problems
transcend national and
agency boundaries

13. • Climate change
• Economic turbulence
• Refugees
• Virulent infectious diseases
• Dying oceans
• Global cyber-terrorism
• International drug cartels
• Human trafficking
• Weapons trafficking
• Poverty and inequality
• Multi-national corporate collusion

14. Global problems….
• The definitions of the problems are disputed
• The “facts” are a matter of intense debate
• Politics and special interests dominate:
-- national interests
-- multi-national corporate interests
-- agency agendas
-- competition for resources
• The stakes are huge

15. Evaluation grew
up in the
“projects”
Evaluation grew up in the
projects

16. Evaluation
Traditional
Evaluating…
• Grants
• Projects & Programs
• Clusters of grants
• Goal attainment
• Outcomes
• Implementation
Generating...
• Findings & Lessons
• Recommendations
Nontraditional &
New Directions:
Evaluating…
• Mission fulfillment
• Strategy
• Innovation
• Advocacy campaigns
• Policy change
• Systems Change
• Complex dynamic
interventions 16

17. Innovations & Challenges:
Evaluating…
• Community impacts
• Regional initiatives
• Networks and collaborations
• Leadership
• Inclusiveness and diversity
• Innovation
• Collective impact
• Scaling
• Environmental ecosystem sustainability
• Global systems change & TRANSFORMATION
17

20. LOCAL
GLOBAL
GLOCAL

21. 21

22. 3M
POST-IT
NOTES,
50
years ago
22

25. From evaluating
fishing
to
evaluating
the sustainability of the fishing cosystem

26. 26

28. From International to
Global

29. Are we ready for….
Evaluators without borders

31. Blue Marble Evaluation
Core Principles

33. Global Thinking
Principle

34. 34
RIGOROUS SCIENTIFIC
THINKING:
❑ Critical thinking
❑ Inferential thinking
❑ Skill in interpreting
evidence
❑ SYSTEMS THINKING

35. Carl Sagan, cosmologist
“Science is a way of thinking
much more than it is a body of
knowledge.”
1934 - 1996

38. “We can not solve our problems with the
same level of thinking that created them.”

40. Blue Marble Global
Scale Principle
Design Guidance Evaluation Criterion
and Question
Engage using a
global perspective.
Look beyond nation-
state borders and
boundaries to affect
transnational,
regional, and global
patterns,
interactions, and
dynamics.
Global scale fidelity:
To what extent and
in what ways is an
initiative or
intervention truly
global in both
processes and
results?
Blue Marble Global Scale Principle

48. Geologic Time
• Earth’s history has been broken up into time units
based on the fossils contained within the rocks. These
units are part of the Geologic Time Scale.
• The time scale is broken up into units called eons,
eras, periods, and epochs

52. Stockholm
Resilience Centre
Great
Acceleration

54. Peter Drucker, management scholar
“The greatest danger in
times of turbulence is not
the turbulence—
It is to act with
yesterday’s logic.”
1909-2005

55. Ecology & Society

63. Overcoming
Boundaries

64. Integral
Whole
Connected
Inter-related
SDGs

65. Interconnected SDGs

66. Grouping SDGs
credit: Azote Images for Stockholm Resilience Centre

67. 67
TRANSFORMATION

71. WHY?

73. Getting to Maybe:
How the World Is
Changed?
Frances Westley,
Brenda Zimmerman,
Michael Q. Patton
73

74. “We are the first
generation to know that
we are destroying the
planet,
and the last
generation that can do
anything about it.”
WWF (2018)

77. “In Scotland, we are
living as if we have the
resources of three
planets when we only
have one. Australia is
consuming five
planets of resources.
For the world as a
whole, we are
consuming the Earth’s
resources as if we
have 1.7 planets. We
only have one.”
Terry A’Hearn,
CEO, Scottish
Environmental
Protection
Agency

78. “Climate change really
does change everything. If
we are going to meet
global targets or stabilize
global temperatures, then
we are going to have
significant change
whether we like it or not.”
Professor Ioan Fazey,
director of the Centre
for Environmental
Change and Human
Resilience, University
of Dundee.

79. Suzanne Moser,
Stanford Woods
Institute for the
Environment
Transformations, If it is life we want

80. Kumi Naidoo,
Launch Executive
Director of the
African Civil
Society Initiative
“The most important challenge for leadership of
our time, whether the leaders are from the
private sector, from government, or from civil
society, is how can we tell it like it is, speak truth
to power, say that we are five minutes till
midnight on climate change.”

81. “I globe trot from one
meeting to another. Some
of us are caught up in this
cycle and there seems to
be no way out of it.
Anyway, this year (2017), I
participated in nine
international meetings. I
was active in all of them
either as part of the
organizing group or as a
presenter. There is one
thread connecting all of
them: ‘transformation.’”
Million Belay is Coordinator of the Alliance for Food
Sovereignty in Africa (AFSA), a broad alliance of
different civil society actors that are advocating for
and working towards food sovereignty and
agroecology in Africa.

82. 82

83. “We believe that transformational
change requires that we craft new and
better solutions at all scales through a
systems-level approach and deep
collaboration between philanthropy,
researchers, grassroots movements,
the private sector, farmers and food
systems workers, Indigenous Peoples,
government, and policymakers.
Transformation means realizing
healthy, equitable, renewable,
resilient, and culturally diverse food
systems shared by people,
communities, and their institutions.”
(https://futureoffood.org/about-us/
Ruth Richardson,
based in Toronto, is
Executive Director of
the Global Alliance
for the Future of
Food

85. Transformation involves redefinition of
goals (e.g.: from producing energy to
producing sustainable energy) which arise
from a new understanding about the way
things work (eg.: carbon emissions result
in climate change) and produce
fundamental change in operating logics
(e.g.: from “mining” of nature, to harmony
with nature). This usually involves deep
shifts in power structures (e.g.: away from
carbon extractors).

86. GEORGE FLOYD: “I CAN’T
BREATHE.”

88. SYSTEMS CHANGE … beyond projects
•Deep
•Broad
•Long-term

89. Understanding the
Elephant
from
a Systems Perspective
89

90. 90

91. 91

92. 92

93. 93

94. 94

95. 95

96. 96

97. 97

98. The relationship between what
goes in and what comes out
What conceptual
framework
informs situation
recognition and
sense-making?
98

99. Principles of Systems Thinking
•Interrelationships
•Perspectives
•Boundaries
•Dynamics

100. Understanding complex,
dynamic interconnections

101. Complexity concepts informing transformation
• Emergence: Self-organizing, Attractors
• Nonlinear: Small actions generate large effects (Butterfly
effect)
• Dynamical: Interactions within, between, and among
subsystems and parts within systems can volatile, changing
• Getting to Maybe: Uncertainty; unpredictable;
uncontrollable; unanticipated consequences
• Coevolution: Interdependence
• Adaptation: Response to turbulence, changing conditions
• Tipping points: Not knowable in advance, clear afterwards
• Critical mass: Multiple pathways converging
101

103. The Water of Systems Transformation

105. End of apartheid

106. Bangladeshi economist Mohammed
Yunus
106

107. Microfinance: Grameen Bank
• “solidarity group lending”
• dividing a loan among several recipients
• limited to women, a stipulation created by Yunus
• He felt that women were more reliable, typically, at
fulfilling their payment obligations and cooperating
with their peers
• women would overwhelmingly invest in their families’
wellbeing
• envisioned microfinance as a vehicle for female
empowerment + material gains.
107

108. 2006

109. Is
transformation
difficult to
measure?
109

110. Interocular
Significance
Sensitizing
Concept

111. Independent
Evaluation
Group,
The World
Bank

112. Dimension Definition
Relevance Strategic advances that accelerate or shift the trajectory of
progress toward low carbon and/or climate resilient development
in targeted countries and sectors.
Scale Catalytic processes that significantly expand and diffuse the
development and deployment of low carbon and climate resilient
technologies, infrastructure, and other innovations, increasing
their supply and access.
Systemic
change
Deep, fundamental shifts in patterns of individual, institutional,
community, and/or private sector decision making, actions, and
behaviors in targeted markets or other systems that advance low
carbon and climate resilient development.
Sustain-
ability
Transformational changes that are designed to be financially,
economically, environmentally, socio-politically, or physically
robust and resilient. These changes are durable and lasting in
ways that lessen the likelihood of reverting back to past practices
and persist over time.
Dimensions of Transformational Change for the
Climate Investment Fund

113. Theory of
Transformation
113

114. Carol H. Weiss
Theory of Change:
“Nothing as Practical
as a Good Theory”
1995 Aspen Institute
Nov. 7, 1926 -
Jan. 8, 2013
114

116. Theory of
Transformation
116

119. Evaluation
Criteria
for
Transformation

121. Michael Scriven. The logic of evaluation.
121

122. The logic of evaluation is a four-step process that
includes:
1) defining the criteria that will be used to evaluate something;
2) setting standards of performance on those criteria;
3) measuring the actual performance; and
4) synthesizing the results to reach an evaluative judgment

123. Michael Scriven. The synthesis problem
123
How to combine the factual premises
with the value premises.

124. Criteria Matter
1. Criteria constitute the nuclear core of evaluation’s energy function:
rendering judgment. Without criteria there can be no judgment. Without
judgment there can be no evaluation.
2. Criteria express, manifest, encompass, make explicit, and operationalize
what is valued. Criteria mediate the conversion of values into judgments.
3. Criteria prioritize what is important.
4. Criteria direct what questions to ask, data to collect, and results to
highlight.
5. Criteria focus evaluation reporting and conclusions: To what extent and in
what ways have criteria been met?

125. OECD/DAC criteria, 1991
• Relevance (Project addresses priorities)
• Effectiveness (Project outcomes)
• Efficiency (Project cost-benefit)
• Impact (Results)
• Sustainability (continuity)

128. DAC Revised Criteria

131. GLOBAL
EMERGENCY

132. The traditional, normal practice of
evaluation
Footings:
Language of problem
and solution
(versus systems and
adaptations)
Project-oriented and
solution-focused
(versus
transformation-driven
and adaptation
oriented)
Assumption of a ‘right’
answer
(versus accepting that
inquiry and learning
are ongoing)
Oriented to goal
achievement
(vs systems
trransformation)
Assumes solutions
that can be
engineered
(versus embracing
complex systems
thinking)
Evaluation Roundtable April 2019 132

133. Post-
Normal
Evaluation
Thomas
Schwandt

134. Peter Drucker

135. DAC Revised Criteria

137. Peter Drucker
“The greatest
danger in times
of turbulence is
not the
turbulence – it is
to act with
yesterday’s
logic.”

138. SUSTAINABLE
SYSTEMS
TRANSFORMATION
EVALUATION
CRITERIA

142. Greenwashing
Disinformation disseminated by an organization so as to present
an environmentally responsible public image.
Greenwashing is the process of conveying a false impression or
providing misleading information about how a company’s
products are more environmentally sound. Greenwashing is
considered an unsubstantiated claim to deceive consumers into
believing that a company's products are environmentally
friendly.
Greenwashing is a play on the term
"whitewashing," which means using
misleading information to gloss over bad
behavior.

143. Transformation
Glossing

144. SUSTAINABLE SYSTEMS TRANSFORMATION
EVALUATION CRITERIA
2. Complex Dynamic Systems Focus
-- Systems are clearly the evaluand
-- Natural and human system(s) are targeted as
interrelated and interdependent
-- Complexity theory and understandings

146. Seeing Through A
Complexity Lens
“You don't see something until you have the
right metaphor to let you perceive it”. Thomas Kuhn
146

147. Inspired by Jeff Conklin,
cognexus.org
Time
Complex development situations are ones
in which this…
147

148. 148
And this…

149. Turns out to be this…
Time
149

150. 150
…looks like this

152. SYSYTEMS COMPLEXITY
• Interconnections
• Perspectives
• Boundaries
• Dynamics
• Emergence
• Nonlinearities
• Uncertainty
• Uncontrollable
• Adaptation
Beyond Projects and Programs

155. SUSTAINABLE
SYSTEMS TRANSFORMATION
CRITERIA
3. Resilient Ecological Sustainability.
Evaluate transformational sustainability as manifesting ecosystem resilience
and adaptability at the nexus between humans and the environment.
• Employ a dynamic view of sustainability.
• Make the ecosystem viability the focus of sustainability not a
program, project, or intervention.
• Not sustainability as continuity

156. DAC Revised Criteria

159. The International Development Evaluation Association held its 2019 conference in Prague. The theme was Evaluation
for Transformative Change. The conference concluded with a “Declaration” of 10 recommendations, one of which
addressed sustainability as a universal evaluation criterion:
In all our evaluations, we commit to evaluating for social, environmental and economic
sustainability and transformation…. (IDEAS, 2019)
SUSTAINABILITY as a universal
criterion in evaluations

161. 2015 2019

162. • The Canadian Evaluation Society (CES) is revising its
strategic plan and making environmental
sustainability a guiding principle.
• The 2020 conference organizing committee has also
selected environmental sustainability as one of its
foundational values.

163. 4. Eco-efficient Full Cost Accounting
• Document and assess the full costs and benefits of systems
transformations, including economic, social, and environmental
dimensions.
• Compare the full costs and benefits of baseline versus
transformed systems. Evaluate whether, how, and to what extent
transformational engagement generates net eco-efficient benefits.

164. EXTERNALITIES

165. Business Roundtable
Redefines the Purpose of a
Corporation to Promote
“An Economy That Serves All.”

166. 25 Years Ago I Coined the
Phrase “Triple Bottom
Line.” Here’s Why It’s Time
to Rethink It.
HARVARD
BUSINESS
REVIEW
John
Elkington

170. Ecoefficiency
Creating and delivering goods and services while using
renewable resources and creating less waste (zero) and pollution
(zero).
Full cost accounting
TEEB: The economics of ecosystems and biodiversity

173. 173

174. GLOBAL ALLIANCE FOR THE FUTURE OF
FOOD

175. 5. Diversity/Equity/Inclusion (DEI).
• Evaluate how transformational engagement manifests the values of
diversity, equity, and inclusion.
• Evaluate whether, how, and to what extent transformational
engagement enhances systems level diversity, equity, and inclusion.

178. Oxfam: World's 22 richest men have more than all
women in Africa

179. Oxfam, January 2020, Davos
• The number of billionaires has doubled in the past decade
• At least 119 billionaires worth about $500 billion attended Davos this
year, Bloomberg News reported.
• 2,153 billionaires now have more wealth than the 4.6 billion poorest
people on the planet.

180. VALUES: DEI • Diversity
• Equity
• Inclusion

181. All evaluations should
address equity
Jara Dean-Coffey

182. 182
EQUITY

184. • The Diversity, Equity & Inclusion Working Group (DEIWG)) is twofold:
• to advise the Board on issues relating to diversity, equity and
inclusion within CES, and
• to promote inclusive evaluation practice within the evaluation
community, in reflection of the CES’s revised competencies and
ethical principles.

185. 6. Interconnectedness momentum
Identify, understand, and evaluate the interconnections that are
essential and integral to transformation.
• Evaluate whether, how, and to what extent interconnections
among people, networks, institutions, ideas, and movements are
deepened and enhanced to support, nurture, catalyze, and
accelerate transformational trajectories.
• Evaluate whether, how, and to what extent dysfunctional and
constraining interconnections are disrupted and broken to liberate
positive transformational energy and momentum.

187. “When everything is connected to everything else, for better or
worse, everything matters.”
— Bruce Mau
Canadian designer and educator
Evaluation matters

188. Interconnected SDGs

189. TRANSFORMATION
INTERCONNECTIONS
GLOCAL IMPACTS: Micro-meso-macro interconnections
LOCAL GLOBAL

190. Think globally
Act locally & globally
Evaluate interactions
TRANSFORMATION

191. Global
systems

192. 192

193. Interconnections
across
time
SENSE of URGENCY

194. Posterity – not History, nor the Future
• This is part of a new ethics for this era. It requires
imagining and ‘designing’ futures fully cognisant of
the people who will inhabit them. Evaluation
professionals use the past to help inform the
future. But do our evaluation questions and
criteria reflect a clear notion of the responsibility
of this generation to future generations, the need
to break from the past – and hence the
stewardship that is needed to ensure sustainable
development ‘for posterity’?

195. To the 7th
generation

196. Criteria for Evaluating Transformation
1. Transformation Fidelity
2. Complex adaptive systems focus
3. Resilient adaptive
capacity
4. Ecoefficiency (full cost accounting)
5. Diversity, EQUITY, Inclusiveness
6. Interconnectedness momentum

197. DAC Technical
Criteria
• Relevance (project
addresses priorities)
• Effectiveness (project
outcomes)
• Efficiency (project cost-
benefit)
• Impact (Project results)
• Sustainability (continuity)
• Coherence
Transformation Values-
Based Criteria
• Transformation Fidelity
• Complex adaptive
systems focus
• Resilient adaptive
capacity
• Ecoefficiency (full cost
accounting)
• Diversity, EQUITY,
Inclusiveness
• Interconnectedness
momentum

200. Your questions & comments