In this millennium the worldwide web has enabled new models of collaboration and the power of networks to emerge. In the second decade of the new millennium these ideas continue to spread. Cross-disciplinary teams, open innovation and social networks represent radically different approaches to working in systems to create knowledge, share information and develop interventions. Think Wikipedia. Methods for program planning and evaluation need to keep pace with these changes and concept mapping methodology may have been ahead of its time as a method that resonates with 21st century complexity. To think prospectively, this session will reframe concept mapping as a method that employs open innovation and networks to create meaning about complex phenomena. With this basis, this session will explore through presentation and discussion the future possibilities for the types of problems that can be addressed and ways to co-create meaning with diverse stakeholders.

1. And Then the Internet Happened
Prospective Thoughts about Concept Mapping in the New Millennium
Panel Session 599
Concept Mapping at 25:
Development, Application, and Future Directions
Columbia Section 2
Friday, Oct 18, 2:40 PM to 4:10 PM
Presentation at the annual conference of the American Evaluation Association
Washington, DC
William Trochim, PhD
Professor, Policy Analysis & Management
Cornell University
Daniel McLinden, EdD
Assistant Professor, General & Community Pediatrics
Cincinnati Children’s Hospital Medical Center
Special thanks
Melida D. Busch, MLIS,
Manager, Edward L. Pratt Medical Research Library
Cincinnati Children’s Hospital Medical Center

2. Predicting the future is easy - Getting it right is difficult
“The future is now, and we are
not impressed… Now is the time
to stop wishing, to stand up, and
to shout, "Where the hell is my
jetpack?!”1
1 Wilson, DH. (2007). Dude, Where's My Jetpack? A look at the future that refuses to arrive. Discover, February, accessed
online at http://discovermagazine.com/2007/feb/jetpack-future-technologies
2 Steve Jobs as quoted in BusinessWeek (12 May 1998) accessed online at
http://www.businessweek.com/bwdaily/dnflash/may1998/nf80512d.htm.
3 xkcd accessed online at : http://xkcd.com/864/
“…people don't know
what they want until
you show it to them.” 2

3. The internet happened and it got big!
• Back in the day
– In 1989, while working at CERN, Tim Berners-Lee invented a network-
based implementation of the hypertext concept.4
– The first commercial dialup ISP in the United States …opened in 1989.4
– Websites in 1990 – one.5
– Concept mapping illustrates emergent internet properties of the future in
1989.
• More recently
– As of today, we see a total of 147,262,819 domains across the .COM, .NET,
.ORG, .INFO, .BIZ, and .US top level domains (TLDs).1
– The Indexed Web contains at least 4.32 billion pages.2
– One trillion unique URLs3
– Internet gives rise to new models of collaboration and methods of
studying collaboration.
1 "Domain Counts & Internet Statistics". Accessed online at: http://www.whois.sc/internet-statistics/.
2 "The size of the World Wide Web". Accessed online at: Worldwidewebsize.com.
3Alpert, Jesse; Hajaj, Nissan (25 July 2008). Accessed online at: "We knew the web was big...". The Official Google Blog.
4 History of the internet. Accessed online at: http://en.wikipedia.org/wiki/History_of_the_Internet#cite_note-89.
5 How we got from 1 to 162 million websites on the internet. Accessed online at: http://royal.pingdom.com/2008/04/04/how-we-got-
from-1-to-162-million-websites-on-the-internet/.

4. In order to increase the probability of being correct,
let’s stand here and predict the future
"Who Let the Dogs Out,"
Baha Men
2000
The New York Yankees defeat
the New York Mets in Game 5
of the 2000 World Series
Gisbourne, New
Zealand population
32,754 is first city in the
world to welcome in
the new millennium

5. Looking Back
Agreement and consensus is the road that leads to the nowhere

6. Looking forward, concept mapping enables diverse thinking

7. 1 Wikipedia at http://en.wikipedia.org/wiki/Crowdsourcing
2 Howe, J. (2006) ‘The rise of crowdsourcing’, Wired, vol. 14, no. 6. Accessed online at
http://www.wired.com/wired/archive/14.06/crowds.html
3 Howe, J. (2008) Crowdsourcing: Why the Power of the Crowd is Driving the Future of Business, Crown: New York,
NY.
4 Benkler, Y. (2011). The Penguin and the Leviathan: The triumph of cooperation over self-interest. New York:
Crown Business.
“Crowdsourcing is the practice of obtaining needed services, ideas, or
content by soliciting contributions from a large group of people.”1

8. “…the more power you give a single individual
in the face of complexity and uncertainty, the
more likely it is that bad decisions will get
made... (p. 220)”
Surowiecki, J. (2004). The Wisdom of Crowds. New York: Doubleday
1. diversity of points of view,
2. independence from each other’s opinions,
3. some kind of mechanism for aggregating
private judgments into a collective decision.
Tapping into the wisdom of crowds requires

9. 1. diversity of points of view,
2. independence from each other’s opinions,
3. some kind of mechanism for aggregating private judgments into a collective decision.
Surowiecki, J. (2004). The Wisdom of Crowds. New York: Doubleday
1. “…conceptualization is best when it includes a wide variety of relevant people (p.2).”
2(a). “… there should be no criticism or discussion regarding the legitimacy of statements
which are generated…also allow each participant to submit several statements
anonymously on paper so that confidentiality will be preserved (p.4-5).”
2(b). sort the cards into piles “in a way that makes sense to you (p.5).”
3 (a). “…conduct a two-dimensional nonmetric multidimensional scaling… (p.7).”
3 (b). “…hierarchical cluster analysis … to group individual statements on the map into
clusters of statements which presumably reflect similar concepts (p.8).”
Trochim, W. M. K. (1989). An introduction to concept mapping for planning and evaluation. Evaluation and Program
Planning, 12(1), 1–16.
* Emphasis added
Similarities between now and then

10. Simple problems & simplifying problems.
Over Under
• 72% – hang toilet paper with the first
sheet going over the roll.
• 28%- prefer to hang the first sheet
under the roll.
“This is not rocket science.”
Which is correct?
Source: Many internet sites that are
devoted to reporting this important and
fascinating result.
“It is much more difficult
than Rocket Science.”
Source: Me in response to
stakeholders wanting to
oversimplify a complex issue.

11. Looking forward, I predict an appreciation of
wicked problems & simple but profound insights
1 Rittel, HWJ & Webber, MM. (1973). Dilemmas in a general theory of planning. Policy Sciences,
4, 155-169.
2 Attributed to Thomas Schelling, date unknown.
“…wicked in a meaning akin to … vicious…or tricky…it
becomes morally objectionable for the planner to
treat a wicked problem as though it were a tame one,
or to tame a wicked problem prematurely, or to refuse to
recognize the inherent wickedness of social problems
(160-161).” 1
“One thing a person cannot do, no matter how rigorous
his analysis or heroic his imagination, is to draw up a list
of things that would never occur to him.” 2
Emphasis added.

12. 1Burke, JG, O’Campo, P, Peak, GL, Gielen, AC, McDonell, KA, & Trochim, WMK. (2005). An introduction to
concept mapping as a participatory public health research method. Qualitative health research, 15(10), 1392-
410.
2 Rosas, SR. (2012). The utility of concept mapping for actualizing participatory research. Cuadernos
Hispanoamericanos De Psycologia, 12(2), 7-24.
Concept mapping as a participatory method1,2 addressing
wicked problems
Thinking together with diverse perspectives

13. • characteristics of your school environment that could relate
in any way, good or bad, to a student’s experience of
violence.1
• characteristics of neighborhoods that could relate in any way,
good or bad, to women’s experience of intimate partner
violence.2
• role of drugs and alcohol in Newark’s low-income and
predominantly African American neighborhoods3
• ways to deal with the problem of bullying in children4
• ways to deal with the problem of overweight/obesity in
children 4
1 Johnson, SL, Burke, JA, & Gielen, AC. (2011). Prioritizing the school environment in school violence prevention efforts. Journal
of School Health. 81, 331-340.
2 O’Campo, P., Burke, J., Peak, GL, McDonnell, KA, & Gielen, AC. (2004). Uncovering neighbourhood influences on intimate
partner violence using concept mapping. Journal of Epidemiology and Community Health. 59(7), 603-608.
3 Windsor L.C. (2013). Using Concept Mapping in Community-Based Participatory Research: A Mixed Methods Approach. Journal
of Mixed Methods Research, 7(3), 274-293.
4 Vaughn, L.M.; Jacquez, F. & McLinden, D. (2013). The use of concept mapping to identify school-driven intervention strategies for
physical and mental health. Health Promotion Practice. 14(5), 675 - 685.
For example,

14. Looking forward I predict that participatory will expand to include youth
1 Borden, L., Perkins, D., Villarruel, F., Carleton-Hug, A., Stone, M., & Keith, J. (2006). Challenges and opportunities to Latino youth development: Increasing meaningful participation in youth development programs. Hispanic Journal
of Behavioral Sciences, 28(2), 187–208.
2 Chun, J. & Springer, D.W. (2005). Stress and coping strategies in runaway youths: An application of Concept Mapping. Brief Treatment and Crisis Intervention, 5(1), 57-74.
3 Davis, T. S., Saltzburg, S., & Locke, C.R. (2010). Assessing Community Needs of Sexual Minority Youths: Modeling Concept Mapping for Service Planning. Journal of Gay & Lesbian Social Services, 22(3), 226-249.
4 Johnson, S.L., Burke, J.G., Gielen, A.C., (2011). Prioritizing the School Environment in School Violence Prevention Efforts. Journal of School Health, 81(6), 331-340.
5 Ridings, J.W., Powel, D.M., Johnson, J.E., Pullie, C.J., Jones, C.M., Jones, R.L., et al (2008). Using concept mapping to promote community building: The African American initiative at Roseland. Journal of Community Practice, 16(1),
39-63.
6 Ries, A.V., Voorhees, C.C., Gittelsonhn, J., Roche, K.M., & Astone, N.M. (2008). Adolescents' Perceptions of Environmental Influences on Physical Activity. American Journal of Health Behavior, 32(1), 26-39.
7 Vaughn, L.M.; Jacquez, F. & McLinden, D. (2013). The use of concept mapping to identify school-driven intervention strategies for physical and mental health. Health Promotion Practice. 14(5), 675 - 685.
“(a) One of the reasons young people take part in youth
programs is ___” and (b) “One of the reasons other
young people are NOT involved in youth programs is ___
(191).”1
“…young people between the ages of 9 and 19 who
participated in some type of organized youth program and
identified their ethnicity as Latino (191).” 1
“Something GLBT youths need from their communities to
feel supported is ... (231).” 3
“…included youths ages 14–23 … identified their sexual
orientation as 55% lesbian, 35% gay, and 10% bisexual
(230).” 3
“…generate a list of items that describe characteristics
of your school environment that could relate in any way,
good or bad, to a student’s experience of violence (333).’’4
“…10th and 11th grade…” and “..9th and 10th
grade…(332)”4
“The youth were recruited from both high school and
elementary levels…(44).” 5
“Please tell us about a problem or issue facing young
African American males between the ages of 10 to 16 in
the Greater Roseland Community…(47)” 5
“…9th through 12th graders from 2 magnet high schools
located in Baltimore…(26).” 6
"What things in the environment, both good and bad,
might influence physical activity among adolescents
(27)?“6
“…runaway youths …ages of 13 and 18 years, had left their
homes without permission for more than 24 hours, and
currently lived in a runaway shelter (61)” 2
“… Things that stress me out are ___ …” and “…The
way I deal with stress is to ____ (60).” 2
“…college students and school stakeholders (students,
parents, teachers, administrators) (677).” 7
“Thinking of your school, list the ways to deal with the
problem of bullying in children… ((677).” 7

15. In the future, I speculate*…
• We can sort more than
text?
• Visualizing & analyzing will
evolve.
* I am less sure and more worried I may be predicting jet-packs
and flying cars.

16. Picture this…
Can we do more of this? Sort other items to make meaning?
“Seventy-seven photographs with 1-
to 2-sentence captions providing a
context for the photograph were
used as input and served as the
foundation for the sorting and rating
procedures (196)”
Haque, N & Rosas, S. (2010). Concept mapping of photovoices: Sequencing and integrating
methods to understand immigrants’ perceptions of neighborhood influences on health. Family
& Community Health, 33(3), 193-206.
Accessed online at: http://www.wellesleyinstitute.com/wp-content/uploads/2012/07/Report-
20090901-Integration-of-Concept-Mapping-and-Photovoice.pdf

17. Visualizing hard art & maybe not so soft science
But perhaps most of all, you may
find that some of the maps are
simply “pretty” and interesting
to look at (p 88).” 1
1 Trochim, WMK. 1989). Concept Mapping: Soft science or hard art? Evaluation and Program Planning. 12, 87-
110.
2 Trochim, WMK. (1989). Outcome pattern matching and program theory. Evaluation and Program Planning.
12, 355-366.
“pattern matching approach implies a
different view of data … it treats relevant
data about programs, measures,
participants, or outcomes as patterns or
as a whole rather than just as a collection
of individual measures or observations (p
358).”2
Emphasis added
Provide a Greater Community
and Global Perspective
Pay Attention to
Presentation Design
Impact Primary Care
Be Accessible through
Multiple Channels
Utilize Outside/Visiting
Presenters
Highlight CCHMC Focus on Innovation
Include Topics for a
Diverse Audience
Cover Current Practice
A B A B

18. Visualizing hard art & maybe not so soft science
But perhaps most of all, you may find
that some of the maps are simply
“pretty” and interesting to look at
(p 88).” 1
1 Trochim, WMK. 1989). Concept Mapping: Soft science or hard art? Evaluation and Program Planning. 12, 87-110.
2 Trochim, WMK. (1989). Outcome pattern matching and program theory. Evaluation and Program Planning. 12, 355-366.
“pattern matching approach implies a different
view of data … it treats relevant data about
programs, measures, participants, or outcomes
as patterns or as a whole rather than just as a
collection of individual measures or observations
(p 358).”2
Emphasis added

19. What goes around comes around,
visualizing program theory
1 Trochim, W. M. K. (1989). An introduction to concept mapping for planning and evaluation. Evaluation and
Program Planning, 12(1), 1–16.
In 1989: “Group concept mapping is consistent
with the growing interest in the role of theory in
planning and evaluation (p 2).” 1
Logic Model for TTURC-I Evaluation accessed online at: https://www.teamsciencetoolkit.cancer.gov/public/TSResourceTool.aspx?tid=1&rid=762
See also: Trochim, WM, Marcus, SE, Masse, LC, Moser, RP, & Weld, PC. (2008). The evaluation of larger research initiatives: A participatory
integrative mixed –methods approach. American Journal of Evaluation, 29(1), 8-28.
Stokols D., Fuqua J., Gress J., Harvey R., Phillips
K., Baezconde-Garbanati L., Unger J., Palmer
P., Clark M.A., Colby S.M., Morgan G., Trochim
W. (2003). Evaluating transdisciplinary science.
Nicotine and Tobacco Research. 5(SUPPL. 1),
S21-S39.
Yampolskaya, S., Nexman, TM, Hernandez, M., &
Koch, D. (2004). Using concept mapping to
develop a logic model and articulate program
theory: A case example. American Journal of
Evaluation, 25(2), 191-207.
Trochim, W., Milstein, B., Wood, B. J., Jackson,
S., & Pressler, V.(2003). Setting objectives for
community and systems change: an application
of concept mapping for planning a statewide
health improvement initiative. Health Promotion
Practice, 8, 1-12.
In the new millennium, The logic
model connection to concept mapping:

20. What does a group of people think
about an issue, a problem, an
opportunity?
Concept mapping
How does a thought (or other
things) travel among a group of
people?
Social Network Analysis
The rise of network of analysis will influence concept mapping.
See for example,
Christakis, NA & Fowler, JH. (2009). Connected: The amazing power of social networks and how they shape our
lives. New York, NY: Little, Brown & Co.
Durland, M. M. & Fredricks, K. A. (Eds.), (2005a). New directions for evaluation: Social network analysis in program
evaluation: San Francisco, CA: Jossey-Bass.
Gloor, P.A. (2006). Swarm Creativity. Oxford. Oxford University Press.

21. 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18 19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46 47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63 64
65
66 67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
Practice change
Broad
Perspective
Multidisciplinary
Diverse audience
Access/format
Visiting presenters
Organization-centric
Innovation
Inspiration
This is a concept map

22. See for example, McLinden, D. (2013). Concept maps as network data: Analysis of a concept map using the
methods of social network analysis. Evaluation and Program Planning, 36(1), 40-48.
This is a network analysis of a concept map
Finding additional meaning
* Dichotomized the
sorting matrix, GE 11=
1, LT 11=0
Practice change
Broad
Perspective
Multidisciplinary
Diverse audience
Access/format
Visiting
presenters
Organization-centric
Innovation
Inspiration

23. A little more network analysis of a concept map.
What more could we do?
Practice change
Broad
Perspective
Diverse audience
Access/format
Visiting presenters
Organization-centric Innovation
Inspiration
Multidisciplinary
* Increasing point size
corresponds to increasing
degree centrality
See for example, McLinden, D. (2013). Concept maps as network data: Analysis of a concept map using the
methods of social network analysis. Evaluation and Program Planning, 36(1), 40-48.

24. I speculate that visualization will increase
Also see, Borner, K. (2010). Atlas
of science: Visualizing what we
know. Cambridge, MA: MIT Press.
Check out Jove, the first
scientific video journal
http://www.jove.com/
This is a spherical plot of a concept map using
SmacofShpere in R, Stress=0.045.*
Could 3D visualization make a
comeback?
Should we emulate a trend in the
physical sciences?
* R Core Team (2013). R: A language and environment for statistical computing. R Foundation for Statistical
Computing, Vienna, Austria. URL http://www.R-project.org/.

25. Back to the future
In the new millennium, concept mapping has provided an approach to:
• Address wicked problems
• Enable collaborative & participatory decision-making
• Embrace diverse thinking
• Visualize the issues of a community
What more can we do to further the art and science?
xkcd accessed online at: HTTP://XKCD.COM/209/

26. And Then the Internet Happened:
Prospective Thoughts about
Concept Mapping in the New
Millennium
Contact information:
Daniel McLinden, Ed.D.
Cincinnati Children's Hospital Medical Center
3333 Burnet Avenue
ML 3026
Cincinnati, OH 45229-3039
Office: 513 636 8933
Mobile: 513 739 9087
Email: daniel.mclinden@cchmc.org
Skype: danmclinden
Twitter: danmclinden