"On Tour in the Garden of Empirical Analysis" The authors report findings, both solid and speculative, after six years of an ongoing effort to collect, organize, and understand all available empirical research on the use of games to create environments in which people learn.

1. On Play in the Garden of
Empirical Analysis
Serious Play Conference
20-22 August 2013
Redmond, WA
J. D. Fletcher
Institute for Defense Analyses
Sigmund Tobias
State University of New York at Albany
fletcher@ida.org

2. Synthetic Environments:
E.g., Simulations & Games

3. On Simulations and Games: Similarities
Simulations Games
Synthetic environments Synthetic environments
Missions Missions
Successful mission
completion despite
competition
Successful mission
completion despite
competition
Highly interactive Highly interactive
Governed by rules of
engagement
Governed by rules of
engagement
Selective realism Selective realism

4. On Simulations and Games: Differences
Simulations Games
Will sacrifice entertainment
in favor of reality
Will sacrifice reality in favor
of entertainment
Scenario/realistic tasks Storyline/fictional quests
Emphasis on task
completion
Emphasis on competition
and levels
Not necessarily interactive Necessarily interactive
Focus on (rule)
accuracy/detailed
Focus on (rule)
clarity/stylized
Not all simulations are
games
All games are simulations

5. Synthetic Environments:
Advantages
• Safety
• Economy
• Visibility
• Time Control

6. 1909 Antoinette Simulator 1910 Saunders Teacher
1930 Link Trainer ?? Today

7. Top Gun: An (Unintentional) Warfare Experiment
Air-to-AirCombatExchangeRatio
EnemyLosses/U.S.Losses
2:1
4:1
6:1
8:1
10:1
12:1
1965-1968 1969 1970-1973
USN
Top
Gun
School
Forme
d
No
Air-to-
Air
Action
USN
2.4:1
USAF
2.4:1
USAF
2.4:1
USN
12:1

8. Where Do Synthetic Environments Fit
In?
LearningContent
Learning Objectives
Facts
Simple
Concepts
Adaptive
Procedures
Abstract
Concepts
Remember Understand Apply Analyze Evaluate Create
Simple
Procedures
(Framework courtesy of Anderson & Krathwohl, 2001)
Learning environments
we have known how to
create since the 1960s
with drill and practice.
Learning environments we
are learning how to create
with
authentic, situated, syntheti
c environments.

9. Assessment

10. Level Description Evaluation
(Did we do things right?)
1 Surveys Impressions and opinions?
2 Outcomes Were the objectives achieved?
(Did we do the right things?)
3 Transfer
Did the instruction improve
workplace performance?
4 Benefits Is the enterprise more effective?
Effectiveness: What Are We Looking For?
Kirkpatrick’s Four Levels of Evaluation
What About Costs, Cost/Effectiveness, Cost/Benefits
Return on Investment, etc.?

11. What We Need: A Wish List
• Agreement on the operational definition of an
instructional game (!)
• Examination of variables that are exclusive to games
compared to robust competitors
• Measures of pre-test and post test learning
• Comparison with a control group receiving robust
instruction intended to produce the same instructional
objective
• Achievement measured quantitatively and in the same
way for both treatment and control groups

12. An Argument for Using Games in Learning
We Know:
• Games can be compulsively motivating and immersing,
• People will voluntarily persist in playing games longer than they
will engage in non-game learning,
• If the game is instructionally relevant, this engagement will
increase time on (learning) tasks,
• Increased engaged time on learning tasks will yield increased
learning.
Therefore:
• People will either learn more from games than from other
instructional environments or learn the same amount at lower
cost.

13. NB: Effect Sizes (Practical Significance)
A descriptive (not inferential) statistic used to estimate the
magnitude of an effect (e.g., experimental treatment). It may
be calculated as:
Cohen’s d =
Mean Group 1 – Mean of Group 2
“Pooled” Standard Deviation
d < 0.20 Negligible
0.20 to 0.39 Small
0.40 to 0.59 Moderate
0.60 to 0.79 Large
d > 0.80 Very Large

14. People Do Learn from Games (1)
For Instance (Perception):
Who What Effect Size
Boot, et al. 2008
Tracking speed 1.83
Visual STM 2.04
Task-switching 1.18
Castel, et al. 2005
Reaction time detect 0.88
Reaction time visual search 1.12
Chisholm, et al. 2010
Response speed 1.31
Speed despite distractor 1.78
Nouchi, et al. 2012
Exec functioning 1.62 & 1.31
Processing speed 1.12 & 1.34

15. People Do Learn from Games (2)
For Instance (Attitude & Affect):
Who What Effect Size
Bachen, et al. 2012
Global Empathy 0.25
Interest in More Learning 0.47
Baldwin, et al. 2010 Self Esteem 0.42
Vos, et al. 2011
Perceived Competence 0.38
Interest 1.44
Effort 1.13

16. People Do Learn from Games (3)
For Instance (Subject Matter):
Who What Effect Size
Gremmen & Potters,
1997
College Economics 0.76
Follow up 0.78
Kebritichi 2010 High School Math 0.39
Ravenscroft &
Matheson, 2002 Grade 1&2 Math 1.70
Segers & Verhoeven
2005
Kindergarten Phonics –
1st grade follow-up
0.43
Suh, & Kim 2010
ESL Listening 0.53
ESL Speaking 0.23
Reading 0.31

17. Alex Wind’s Table
http://www.alexanderpwind.com/all_games_studies.shtml
NB: It’s /all_games_studies.shtml

18. • Randel et al. (1992) in 68 studies:
– 56% of the studies found no difference between
simulations, games, and conventional instruction,
– 32% of the studies favored simulations/games
– 12 of 14 studies found more interest in
simulations/games than classroom instruction
– Greater retention for games/simulations, even in some
studies showing no immediate differences.
– Increasing computer capacity yields more powerful
sophisticated simulations and games
Previous Research Review

19. • Israeli pilots playing Space Fortress II (modified to
simulate cockpit complexity ) performed better in
actual flight than non gamers (Gopher, Weil, & Bareket,
1994)
• But assigning trainees to an off-the-shelf game
(Apache Strike Force) had no transfer effects (Hart
& Battiste, 1992)
A Paradox

20. Conduct Cognitive Task Analysis to Identify the Cognitive
Processes Required by the Game and Task(s) to be
learned.
– It is not just physical similarity between games and
tasks that lead to learning and transfer. It is the
overlap in cognitive processes engaged by both.
Recommendation 1

21. Provide Guidance
a) Pictorial Support.
Reduces cognitive demands of game (cf The Multimedia
Principle)
b) Encourage Participants to Reflect About Correct Answers
Not to reflect about wrong answers
c) Use Unsupported Discovery Learning with Caution
Kirschner, Sweller, & Clark (2006) cite “unambiguous
evidence that minimal guidance during instruction is
significantly less effective and efficient than guidance
specifically designed to support the cognitive processing
necessary for learning” (p. 76).
Recommendation 2

22. Use First Person (“I” “You”)in Player–Game
Dialogue.
–Some findings suggest this practice will increase
learning and transfer
–Unknown whether using player names improves
learning
Recommendation 3

23. Use Animated Agents in Interactions with Players
Some conflicting findings, especially for higher order
learning
Since agents have never been shown to reduce
learning they might as well be used
Use Human, Not Synthetic Voices
Seems to increase learning and transfer
Easy to find good speakers or actors to record speech
Recommendations 4 & 5

24. –Maximize User Interactivity
• Increases learner involvement in game (Flow) and
post test scores
–Reduce Cognitive Load
• Sweller, Mayer, and others have shown importance
of attending to cognitive load
(e.g., present graphics with text)
Recommendations 6 & 7

25. –Maximize Motivation
• Games should have appropriate challenge,
arouse curiosity, & include fantasy elements
appealing to users
(Yerkes-Dodson Inverted U; Vygotsky’s ZPD)
• Assure that motivation is related to game
success and avoid seductive details that
reduce learning
Recommendation 8

26. –Increase Pro-social Content & Reduce Aggression
(per Gentile, 2009)
• Games can increase pro-social attitudes
• Games can increase aggression
• Hence minimize aggressive game behavior and
maximize pro-social content increased
Recommendation 9

27. –Revise Games and Task Analyses
• Off the shelf games that appear similar to task may
not emphasize same cognitive processes
– Integrate Games with Instructional and Task
Objectives
• Research indicates that games not integrated into
curriculum/task lead to learning game strategy but
not more general learning
Recommendations 10 & 11

28. –Keep Abreast of Research Findings
• Knowledge in area is exploding & recommendations
may be revised in light of newer findings.
–Use Teams to Develop Instructional Games.
• No one person has skills/knowledge in cognitive task
analysis, game design, computer expertise,
familiarity with research findings
Recommendations 12 & 13

29. …There is nothing more difficult to take
in hand, more perilous to conduct, or
more uncertain in its success, than to
take the lead in the introduction of a
new order of things...
Niccolo Machiavelli (1469 – 1527), De
Principatibus, 1513
About Change …

30. There is no sun without shadow, and it is
essential to know the night … The struggle
itself toward the heights is enough to fill a
man’s heart. One must imagine Sisyphus
happy.
-- Albert Camus
It’s all rock and roll to me.
-- M. Jagger & K. Richards
And Finally

31. Questions? Comments?
Objections? Complaints?
Thank you!

32. Questions? Comments?
Objections? Complaints?
Thank you!