Keynote presentation of Yannis Dimitriadis at Intelligent Tutoring Systems 2022: Human-Centered Learning Analytics: Designing for balanced human and computational agency

1. Human-Centered Learning
Analytics: Designing for balanced
human and computational agency
Prof. Yannis Dimitriadis
GSIC/EMIC group
University of Valladolid, Spain
Intelligent Tutoring Systems 2022
July 1, 2022

2. Lots of acknowledgements
2

3. Learning Analytics (LA)
in Technology Enhanced Learning (TEL)
Learning Analytics
“measurement, collection, analysis and reporting of data
about learners and their contexts, for purposes of
understanding and optimizing learning and the
environments in which it occurs”
n Most R&D - Innovation has been devoted to
– Mining patterns
– Deriving predictive models
– Providing dashboards
3

4. Predictive models with LA
(At-risk students)
4
Herodotou, C.; Hlosta, M.; Boroowa, Avinash; R., Bart; Zdrahal, Z. and Mangafa, C. (2019). Empowering online
teachers through predictive learning analytics. British Journal of Educational Technology, 50(6) pp. 3064–3079.
How does the predictive model work and how it was trained?
How were the data collected for this prediction model?
Who was involved in its design and who can use the data?

5. Pattern mining using LA
(Detection of learning strategies)
5
J. B. J. Huang, A. Y. Q. Huang, O. H. T. Lu and S. J. H. Yang, "Exploring Learning Strategies by Sequence Clustering
and Analysing their Correlation with Student's Engagement and Learning Outcome," 2021 International Conference
on Advanced Learning Technologies (ICALT), 2021, pp. 360-362, doi: 10.1109/ICALT52272.2021.00115.
How are the proxies for strategies DEFINED AND COMPUTED?
Who can interpret this data and how?
Is there any student bias regarding these strategies?

6. LA-based dashboards
(Monitoring and sense-making)
6
S. Charleer, A. V. Moere, J. Klerkx, K. Verbert and T. De Laet, "Learning Analytics Dashboards to Support Adviser-
Student Dialogue," in IEEE Transactions on Learning Technologies, vol. 11, no. 3, pp. 389-399, 1 July-Sept. 2018,
doi: 10.1109/TLT.2017.2720670
How effective is sense-making out of those dashboards?
Do teachers-students need to improve their data literacy?
Can we compensate the sense-making workload?

7. Smart Learning Environments
(Personalized recommendations-resources)
7
S. Serrano-Iglesias, E. Gómez-Sánchez, M. L. Bote-Lorenzo, G. Vega-Gorgojo, A. Ruiz-Calleja and J. I. Asensio-Pérez,
"From Informal to Formal: Connecting Learning Experiences in Smart Learning Environments," 2021 International
Conference on Advanced Learning Technologies (ICALT), 2021, pp. 363-364, doi: 10.1109/ICALT52272.2021.00116.
How is the student model built?
Do teachers/students get involved in the reaction scripts?
What about privacy in informal learning settings?

8. Two dilemmas on Agency (I)
Dilemma 1: Learning Analytics (LA) may be
helpful when embedded in Technology-Enhanced
Learning (TEL) contexts. They are typically
designed by researchers and developers, that best
know about efficiency and effectiveness. But
existing LA solutions mostly ignore teachers as
orchestrators (designers and enactors).
What about teachers’ agency?
8

9. Two dilemmas on Agency (II)
Dilemma 2: Artificial Intelligence (AI) agents that
are using LA may support and eventually maximize
students’ learning but how can they be transparent,
trustful, responsible or ethical?
What about students’ agency?
9

10. What is this talk about
n Discuss the dilemma regarding teachers’
agency when designing and orchestrating LA
solutions
n Analyze models for human-LA complementarity
and teachers’ augmentation
n Formulate design principles for Human-
Centered Learning Analytics (HCLA)
n Illustrate the HCLA approach
10

11. A definition of teachers’ agency
11
Priestley, M., Biesta, G., & Robinson, S. (2015). Teacher agency: What is it and why does it matter? In R. Kneyber & J. Evers (Eds.), Flip
the System: Changing Education from the Ground Up (pp. 134–148). Routledge. https://doi.org/10.4324/9781315678573 (adapted)
Agency entails the capacity of actors to make practical and normative judgments
among alternative possible trajectories of action, in response to the emerging
demands, dilemmas, and ambiguities of presently evolving situations

12. A socio-cultural perspective of
professional agency
12
Eteläpelto, A., Vähäsantanen, K., Hökkä, P., & Paloniemi, S. (2013). What is agency? Conceptualizing professional
agency at work. Educational Research Review, 10, 45–65. https://doi.org/10.1016/j.edurev.2013.05.001 (adapted)

13. Teachers as producers and shapers
13
Jenkins, G. (2020). Teacher agency: the effects of active and passive responses to curriculum change. Australian
Educational Researcher, 47(1), 167–181. https://doi.org/10.1007/s13384-019-00334-2

14. Digital agency
14
Passey, D., Shonfeld, M., Appleby, L., Judge, M., Saito, T., & Smits, A. (2018). Digital Agency: Empowering Equity in
and through Education. Technology, Knowledge and Learning, 23(3), 425–439. https://doi.org/10.1007/s10758-018-
9384-x (adapted)
Control over and adapt to …
Be proactive producers
Be aware of the data
Decide what data is relevant

15. Human-centered design landscape
15
Sanders, E. B. N., & Stappers, P. J. (2008). Co-creation and the new landscapes of design. Co-design, 4(1), 5-18.

16. From User-Centered Design to Co-Design
16
User-centred design Co-creation (co-design)
User
Researcher
Designer
Sanders, E. B. N., & Stappers, P. J. (2008). Co-creation and the new landscapes of design. Co-design, 4(1), 5-18.

17. Human-Centered Learning Analytics
17
What is human-centeredness
in Learning Analytics then?

18. Human-Centered Learning Analytics
18
Human centeredness has been identified in other
fields as a characteristic of systems that have been
carefully designed by:
• identifying the critical stakeholders,
• their relationships, and
• the contexts in which those systems will function
.

19. Human-Centered Learning Analytics
19
HCD should involve:
Inclusion via stakeholder participation in the design process
+
Empathic experiences (particularly when making design
decisions).
Giacomin, J. (2014). What is human centred design? The Design Journal,
17(4), 606–623. https://doi.org/10.2752/175630614X140561854801.

20. Human-Centered Learning Analytics
Human-centered design considered harmful…
“Most items in the world have been designed without the
benefit of user studies and the methods of Human-Centered
Design. Yet they do quite well.”
What Adapts? Technology or People?
Don Norman proposes stronger focus on tasks and activities
Norman, D. A. (2005). Human-centered design considered harmful. interactions, 12(4), 14-19.

21. Human-Centered Learning Analytics
the human centered (not centric)
All the human factors,
social factors and
technology factors
interact together under the
human activity umbrella.

22. Augmented teacher
(Human-AI complementarity)
22
Holstein, K., Aleven, V., Rummel, N. (2020). A Conceptual Framework for Human–AI Hybrid Adaptivity in Education.
In: Bittencourt, I., Cukurova, M., Muldner, K., Luckin, R., Millán, E. (eds) Artificial Intelligence in Education. AIED
2020. Lecture Notes in Computer Science(), vol 12163. Springer, Cham. https://doi.org/10.1007/978-3-030-52237-
7_20
n Augmentation
– Complementary strengths and weaknesses
– Improvement (co-learning) over time
n Goals
– Optimized objective functions + design
decisions
n Perceptions
– Sense, attention, interpretation

23. Augmented teacher
(Human-AI complementarity)
23
Holstein, K., Aleven, V., Rummel, N. (2020). A Conceptual Framework for Human–AI Hybrid Adaptivity in Education.
In: Bittencourt, I., Cukurova, M., Muldner, K., Luckin, R., Millán, E. (eds) Artificial Intelligence in Education. AIED
2020. Lecture Notes in Computer Science(), vol 12163. Springer, Cham. https://doi.org/10.1007/978-3-030-52237-
7_20
n Actions
– Action space, scalability and capacity
n Decisions
– Link perception and action – take effective
pedagogical interventions
n Timing and granularity
– e.g., adaptation by teachers through LA
dashboards, during learn time, regarding a
task

24. Augmented teacher
(Human-centered approach)
24
Holstein, K., & Aleven, V. (2022). Designing for human-AI complementarity in K-12 education. }, ArXiv,
abs/2104.01266
Echevarría, V. Yang, K., Lawrence, L., Rummel, N., Aleven V., (2020). Exploring Human–AI Control Over Dynamic
Transitions Between Individual and Collaborative Learning, In Proceedings of ECTEL 2020
“
n Lumilo project (CMU) on human-AI
partnership in real-world K-12 education
n Co-orchestration (ITS and teachers) of
transitions from individual to group
activities
n Adoption of participatory (human-
centered) approach to design and
development lifecycle

25. Levels of human-centeredness
25
Smuha N.A. (2023). “Pitfalls and pathways for trustworthy Artificial Intelligence in education” in The Ethics of Artificial
Intelligence in Education Practices, Challenges, and Debates, W. Holmes, K. Porayska-Pomsta (Eds). Taylor and
Francis.
n Human in command
– Oversee when and how to use AI/ITS
n Human on the loop
– Participate in design and operation
n Human in the loop
– Get involved in every lifecycle phase

26. Human-Centeredness in MMLA-AIED
26
Kukurova, M. (2022). “Multimodal Learning Analytics in Real-world Practice: A Bridge Too Far?”, Webinar at Spanish
Network of Learning Analytics (SNOLA), May 2022. https://snola.es/2022/05/03/webinar-multimodal-learning-
analytics-in-real-world-practice-a-bridge-too-far-mutlu-cukurova/

27. Some elements to consider
n LA solutions were eventually pushed by new
technological (Data and AI) affordances
n Teachers as designers were not always
considered in complex real-world TEL spaces
n The hybrid AI-human models and their trade-
offs were not fully studied
n Learning theories have not been used
extensively while designing LA solutions
27

28. The complexity of TEL ecosystems
(Hybrid Learning Spaces)
28
Gil, Mor, Dimitriadis & Köppe (2022): Hybrid Learning Spaces, Springer https://doi.org/10.1007/978-3-030-88520-5

29. Design and orchestration
29
Prieto, L. P., Y. Dimitriadis, J. I. Asensio-Pérez, C. K. Looi (2015). “Orchestration in learning technology
research: evaluation of a conceptual framework”. In: Research in Learning Technology 23.0
How to support teachers as designers and reduce/optimize their
orchestration load?

30. Teachers as designers
n Pedagogical knowledge
– Eventually embedded in tools
– Complements / cooperates with the tacit and
explicit knowledge of the teachers
n Teachers
– Are and can serve as designers
– Should participate in the design and
orchestration of the teaching and learning
processes
30
Kali, McKenney & Sagy (2015)

31. The METIS ILDE
LD and orchestration tools
• Villasclaras-Fernández, Hernández-Leo, Asensio-Pérez & Dimitriadis (2013)
• Håklev, Faucon, Hadzilacos & Dillenbourg (2017) - Figure
• Laurillard, Kennedy, Charlton, Wild & Dimakopoulos (2018) - Figure

32. Balancing computer-human agents
32
Sharples, M. (2013). Shared Orchestration Within and Beyond the Classroom. Computers &
Education. 69. 504-506. 10.1016/j.compedu.2013.04.014.

33. Mirroring, Advising, Guiding through LA
33
Soller, A., Martínez-Monés, A., Jermann, P., Muehlenbrock, M. (2005) From Mirroring to Guiding:
A Review of the State of the Art Technology for Supporting Collaborative Learning International
Journal of Artificial Intelligence in Education (ijAIED). 15:261-290

34. Distributed scaffolding
34
• Puntambekar, S. Distributed Scaffolding: Scaffolding Students in Classroom Environments.
Educ Psychol Rev (2021). https://doi.org/10.1007/s10648-021-09636-3
• https://www.imec-int.com/en/research-portfolio/steams: Supporting TEAMS in ambient learning
spaces
Across
1. Tools and social scaffolds
2. Levels (individual, group, and whole class)
3. Time and Contexts

35. A Hybrid human-AI learning model
35
• Molenaar, I. (2021), "Personalisation of learning: Towards hybrid human-AI learning
technologies", in OECD Digital Education Outlook 2021: Pushing the Frontiers with Artificial
Intelligence, Blockchain and Robots, OECD Publishing, Paris,
https://doi.org/10.1787/2cc25e37-en.

36. Human-AI extended model
n Teacher monitors and controls
– the learning design prior to execution
(configuration phase)
– the orchestration of the lesson (runtime)
n Learner monitors and controls learning
– Orientation and planning prior to execution
– Monitoring and control during execution
– Reflection after execution
36

37. Human-AI extended model
Timing and phases
Detect (data) 
Diagnose (technique/algorithm) 
Act (action)
Act components
– LA Perspective
n Inform, Advise, Guide, Recommend
– ITS Perspective
n Step, Task, Curriculum
37

38. Human-AI extended model
n The transitions of control and monitoring have
profound implications for the professional
functioning (agency) of teachers
– Giving up task has positive sides (less time on
correction, more feedback)
– but also, negative sides (less insights and
control).
n This friction cannot be resolved easily but co-
creation processes do allow for a careful
articulation of this friction
38

39. Human-AI extended model
n Static or dynamic balance
– redesign and reconfiguration
– self-, co-, socially shared regulation
n Operators for teachers’ augmentation
– Transparency, agency, explainability, …
39

40. Hybrid Intelligence
40
D. Dellermann, P. Ebel, M. Soellner, J.M. Lerimesiter, “Hybrid Intelligence”, arXiv:2105.00691v1
[cs.AI]
“… the most likely paradigm for the division of labor between
humans and machines in the next years, or probably decades,
is hybrid intelligence. … to try to combine the complementary
strengths of heterogeneous intelligences (i.e., human and
artificial agents) into a socio-technological ensemble. We
envision hybrid intelligence systems, … to accomplish complex
goals by combining human and artificial intelligence to
collectively achieve superior results than each of the could have
done in separation and continuously improve by learning from
each other”

41. Hybrid Intelligence
41
Z. Akata et al., (2020) "A Research Agenda for Hybrid Intelligence: Augmenting Human Intellect
With Collaborative, Adaptive, Responsible, and Explainable Artificial Intelligence," Computer,
53(8), 18-28, doi: 10.1109/MC.2020.2996587
n “… Hybrid intelligence (HI) can go well beyond this by creating systems
that operate as mixed teams, where humans and machines cooperate
synergistically, proactively, and purposefully to achieve shared goals,
showing AI’s potential for amplifying instead of replacing human
intelligence”
n “Collaborative HI: How do we develop AI systems that work in
synergy with humans?
› Adaptive HI: How can these systems learn from and adapt to humans
and their environment?
› Responsible HI: How do we ensure that they behave ethically and
responsibly?
› Explainable HI: How can AI systems and humans share and explain
their awareness, goals, and strategies?”

42. AI and the future of learning
42
Roschelle, J., Lester, J. & Fusco, J. (Eds.) (2020). AI and the future of learning: Expert panel
Report. Digital Promise. https://circls.org/reports/ai-report.
1. Investigate AI Designs for an Expanded Range of Learning Scenarios
2. Develop AI Systems that Assist Teachers and Improve Teaching
3. Intensify and Expand Research on AI for Assessment of Learning
4. Accelerate Development of Human-Centered or Responsible AI
5. Develop Stronger Policies for Ethics and Equity
6. Inform and Involve Educational Policy Makers and Practitioners.
7. Strengthen the Overall AI and Education Ecosystem
Seven recommendations from US expert panel

43. Human-centered and trustworthy AI
43
• Delgado Kloos, C., et al. (2022), H2O Learn - Hybrid and Human-Oriented Learning: Trustworthy and Human-
Centered Learning Analytics (TaHCLA) for Hybrid Education. IEEE Global Engineering Education
Conference, EDUCON 2022,
• HLEG-AI (High-Level Expert Group on Artificial Intelligence) (2019), “Ethics Guidelines for Trustworthy AI:
Requirements of Trustworthy AI,” Available: https://ec.europa.eu/futurium/en/ai-alliance-consultation/guidelines/1

44. Human-Centered Approaches ...
See also “sister” initiatives for Human-Centered
approaches for the design and development of truly
mixed human-AI initiatives for human empowerment
e.g., Recent EU call for funding of A HUMAN-
CENTRED AND ETHICAL DEVELOPMENT OF
DIGITAL AND INDUSTRIAL TECHNOLOGIES 2022
(HORIZON-CL4-2022-HUMAN-02)
44

45. And a few suggestions …
n Bring together LA and Learning Design (LD)
n Consider multiple needs and paths to use LA,
implemented as adaptive (by system/agent) or
adaptable (by users)
n Bring the teacher in the loop and orchestrate LA
with all stakeholders (OrLA)
n Consider the consolidated model for LA
n Adopt human-oriented workflows for LA solutions
n Consider data storytelling and explanatory LA
45

46. LD-based process for LA solutions
46
1 – LA design: LD elements selected as targets for LA solution
2 – LA implementation:
2a. Data from LA targets is analyzed by the LA tool
Resulting LA informs: 2b.) orchestration, 2c.) assessment
Dimitriadis, Martínez-Maldonado & Wiley (2020)

47. Consolidated model for LA
47
• Gasevic, Dawson & Siemens (2015)
• Saint, Gasevic, Matcha, Ahmad & Pardo (2020)
• Gasevic, Kovanovic & Joksimovic (2017) - figure
• Reimann (2016)

48. Orchestrating LA (OrLA)
48
Prieto, Rodríguez-Triana, Martínez-Maldonado, Dimitriadis & Gašević (2019) - figures

49. LATUX workflow for LA solutions
49
• Martinez-Maldonado, Pardo, Mirriahi, Yacef, Kay & Clayphan (2016) - figure
• Holstein, McLaren & Aleven (2019)

50. Datastorytelling and explanatory LA
50
Echeverria, Martinez-Maldonado, Buckingham Shum, Chiluiza, Granda & Conati (2018) - figures

51. HCD principles for actionable LA
solutions
1. Agentic positioning of teachers and other
stakeholders
2. Integration of the learning design cycle and the LA
design process
3. Reliance on educational theories to guide the LA
solution design and implementation
51
Y.Dimitriadis, K.Wiley, & R.Martínez-Maldonado (2021)

52. Illustrative study
52
From Theory to Action:
Developing and Evaluating Learning
Analytics for Learning Design
• K. Wiley, Y. Dimitriadis, A. Bradford, & M. Linn (2020)
• K. Wiley (2020)
• Y. Dimitriadis, K. Wiley, & R. Martínez-Maldonado (2021)

53. An overview of the study
n Design and development of Teacher Action Planner,
a LA tool that supports teachers’ orchestration
actions:
– Grounded on learning theory (Knowledge Integration)
and using the Inquiry Based Learning approach.
– Aligned with the Learning Design (Global Climate
Change and Photosynthesis Units) and platform (WISE)
– Aligned with stakeholders’ needs (OrLA)
– Functional within the constraints of the technical and
learning environments
53

54. DBR research approach
54
Sandoval & Bell (2004) – figure adapted from

55. The role of Theory: KI framework
55

56. LD informed by Learning Theory
56

57. The feature (and data) to focus on
57

58. Validating the usefulness of LA (I)
58
Are the data used to generate the learning analytics
useful for understanding student learning?

59. Validating the usefulness of LA (II)
59
High Number of Attempts Predicts Performance on
Subsequent Explanation Item

60. Optimizing the LD based on LA
60
Optimization: Fuse two steps of the unit

61. Creating a useful LA solution towards
pedagogical action
61

62. LA report
62

63. Evaluating the LA solution
63

64. The Teacher Action Planner (TAP)
64

65. Recommend orchestration actions
65

66. Human-Centered Design of LA
n Eventually the benefits of enhanced agency,
adoption and impact of the LA solutions
overcome the costs of difficult, time and
resource consuming participatory processes
n All the important aspects of learning
(cognitive, metacognitive, affective and social)
are highly sensible and dependent on the
context
66
Buckingham Shum, S., Ferguson, R., & Martinez-Maldonado, R. (2019). Human-Centred Learning
Analytics. Journal of Learning Analytics, 6(2), 1–9. https://doi.org/10.18608/jla.2019.62.1

67. Some take-home messages (I)
n Technology-enhanced learning (TEL) ecosystems
– Especially hard to design and orchestrate (advising)
n Teachers are essential stakeholders
– LD and LA are both about learning and teaching
n Human-Centered design is necessary despite its cost
– Move from “demonstrators in a greenfield” to embedded tools and
practices in authentic contexts
n Tools are necessary to support stakeholders
– Balanced use of AI agents and human expertise and actions through
orchestration technology and distributed scaffolding
n Keep the power of LA-based models
– But complement with explanations, trust, privacy
67

68. Some take-home messages (II)
n LA for understanding and optimizing learning
– oriented to pedagogical interventions based on actionable insights
n LA benefits from
– Data Science, Learning Theory and Design
n LA and LD are intrinsically interconnected
– They should be jointly employed
n Inter-stakeholder communication is essential
– using multiple design techniques and approaches
n Bring the human in the loop
– Through participatory user-centered design processes
n Support teachers (and learners) with
– technological and conceptual tools
68

69. Some HCLA challenges
● Can design processes from other disciplines, such as HCI,
Co-Design and Participatory design, be unproblematically
adopted for HCLA, or do they require adaptation?
● What are the obstacles to the adoption of HCLA design
processes?
● How can the voice of students be taken more into account,
besides the dominant thread of involving teachers?
● What are the lessons learnt from mid-to-long term HCLA
studies and how do they inform the aforementioned topic of
adoption?
● HCLA beyond conventional higher education
● A wider view of human-centeredness
● Human-AI complementarity
69

70. The Future of HCLA?

71. Let’s remember:
Learning Analytics are about
… Learning
… Learners
…Teachers
… Humans
… Society
This is why
Human-Centered Learning Analytics
may be worth considering