This document discusses controlling adaptation in affective serious games based on recognizing a player's emotional state. It describes how emotional state is recognized using physiological measurements like electrodermal activity and blood volume pulse or analyzing facial expressions with convolutional neural networks. Recognized emotions along with playing style and performance are then used to control adaptation of game mechanics, dynamics, and aesthetics through techniques like dynamic difficulty adjustment and tailoring of non-player characters. The goal is to optimize engagement, motivation, and the learning process through adaptive personalization based on real-time tracking of individual player emotions.
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Controlling Adaptation in Affective Serious Games
1. Controlling Adaptation in
Affective Serious Games
Boyan Bontchev, Ivan Naydenov and Ilko Adamov
Faculty of Mathematics and Informatics,
Sofia University „St Kliment Ohridski“, Bulgaria
3. Serious Games (SG)
“A serious game or applied
game is a game designed for
a primary purpose other
than pure entertainment”
(Abt, 1970)
“A serious game is a digital
game in which education is
the primary goal, rather
than entertainment”
(Micheal & Chen, 2006)
SG enable self-controlled,
active and playfully learning
Source: B., V. Terzieva, Y. Dankov: Educational Video Maze Games, Nauka Journal,
No. 1, 2021, pp. 25-33. 3
4. Affective Serious Games
• Affective computing applies computational approaches for
detecting and deliberating induction of human affect in order to
make human-computer interactions more effective and natural;
• Affective computing is applied in serious games by introducing
control over gameplay based on player affect, i.e. emotional state
• Emotion feeling is a neurobiological activity and can be
recognized by:
• self-reports (filling questionnaires)
• observation of body movements or facial expressions
• psychophysiological measurements of:
• EDA (Electro-Dermal Activity)
• BVP (Blood Volume Pulse)
• ECG (electrocardiogram)
• EEG (electroencephalography) and others
4
5. Adaptation in Affective Serious Games
Affective serious games for learning apply a player-centric
adaptation of:
• game mechanics (i.e. rules dictating interactions and
outcomes in the game, behaviors, and control mechanisms)
• game dynamics (such as task difficulty, pace, and time
pressure) such as:
• dynamic difficulty adjustment (DDA) of game tasks
• tailoring non-player characters (NPC) i.e. virtual players
• game aesthetics (i.e., fun components such as sensation,
fantasy, narrative, challenge, fellowship, discovery,
expression, and submission)
The adaptation process is player-centric and tries to tailor some
or all of them for achieving a better, more effective, and
efficient game-based learning process accompanied by a
facilitated engagement and motivation of the player
5
6. smArt adaPtive videO GamEs
for Education (APOGEE)
• The APOGEE project aims at creation of a software platform for generation of smart
adaptive video games (3D mazes enriched with puzzle mini-games of various types)
• Adaptation based on learning outcomes, learning/playing styles, emotional state
Textual content
Graphics
Audio content
Generated
XML +
multi-
media
content
Unity3D API
Unity3D Editor
Didactic
game tasks
Metadata
(XSD)
Connectivity
Editor
Didactic tasks
Property Editor
Maze Editor Maze Builder
Virtual
players
Intelligent Q&A
agents
Online games
Unity3D game
engine
(browser plugin)
Personalization
and adaptation
Virtual players
3D maze
Automatic
game
generation
6
7. Workflow of Game
Adaptation Control
Gameplay Registration
Visualization
Control
Analysis
Emotion State
Registration
EDA/BVP
Measurement
Game
Adaptation
Procedures
DDA and
NPC Behavior
Real-Time
Player
Feedback
through an
Adaptation
Control Panel
Data
Clustering
Data
Classification
Neural
Network
Facial
Expressions
(Video Stream)
Playing
Style Self-
Report
Game
Outcomes
and Efficiency
7
8. We Recognize Player Emotions by:
• Measurement of physiological data such as EDA and BVP;
• Registration of facial expressions in a video stream or photos of player face taken
during the game.
To determine the emotional state (the four main emotions: fun, boring, relaxing,
scary), we use:
• A convolutional artificial neural network (ANN):
• 700,000 neurons
• organized in 17 layers
• distributed within 4 blocks
• MeanShift and Аgglomerative clustering algorithms with emotion labels taken
from ANN;
• Classifiers such as k-nearest neighbor (kNN), Bayesian networks (BNT), decision
trees (DT), linear discriminant analysis (LDA), support vector machines (SVM), etc.
8
9. ANN Training and Validating
• After training and validating ANN with 32 epochs and more than
20,000 images,
• we evaluated our trained convolutional ANN using a test dataset of
about 10,000 images;
• the reported test accuracy is still as high as 54.36%.
9
10. Extract from the Testing Dataset
• Even though we received 54.36% accuracy on the test and, because
of the overfitting, there are some labels classified to another label.
10
11. Prediction Results
• Prediction results showing true label counts (left
bars present the actual count of each emotion) and
predicted label counts (right bars)
• Confusion matrix
for test dataset
11
12. Heart rate data clustering
(K Means, Mean Shift, Agglomerative)
12
13. Clustering of skin conductance data
(KMeans, MeanShift, Aggloerative)
13
14. Conclusions
• Adaptation of gameplay in affective serious computer games is adjusted on the
basis of tracking of individual player emotions recognized in real-time during the
gaming sessions
• Recognizing emotions will apply real player data – both from physiological
registration of EDA and BVP and from photos and video streams of player face
expressions during gameplay.
• Problems with overfitting should be solved in order to increase the accuracy and
minimize the loss for working with testing datasets.
• The results are going to be compared to new ones received from applying various
classifiers and, thus, an optimal solution for inference of player emotional state
will be found.
• The recognized emotions, together with playing style and player performance,
and efficiency of playing a serious game for learning, are going to be applied for
controlling task difficulty and NPC behavior in an optimal way.
14
