This is an updated version of my presentation at LAK12 that includes related research on TEL RecSys and the new LinkedUp project that will address the current challenges we are facing like the lack of datasets, common evaluation framework and real world examples for LA and data supported education.

1. Learning Analytics
and Future R&D Opportunities
Lecture by Hendrik Drachsler at
RWTH Aachen, Germany, June 20, 2012
1

2. Goals of the lecture
LA Framework
Survey
Findings
Conclusions
2

3. A view on Learning Analytics
The Learning
Analytics
Framework
3

4. Greller, W., & Drachsler, H. (to appear). Turning Learning into Numbers. Toward a Generic
Framework for Learning Analytics. Journal of Educational Technology & Society.
4

5. Greller, W., & Drachsler, H. (to appear). Turning Learning into Numbers. Toward a Generic
Framework for Learning Analytics. Journal of Educational Technology & Society.
4

6. Greller, W., & Drachsler, H. (to appear). Turning Learning into Numbers. Toward a Generic
Framework for Learning Analytics. Journal of Educational Technology & Society.
4

7. Greller, W., & Drachsler, H. (to appear). Turning Learning into Numbers. Toward a Generic
Framework for Learning Analytics. Journal of Educational Technology & Society.
4

8. Greller, W., & Drachsler, H. (to appear). Turning Learning into Numbers. Toward a Generic
Framework for Learning Analytics. Journal of Educational Technology & Society.
4

9. Greller, W., & Drachsler, H. (to appear). Turning Learning into Numbers. Toward a Generic
Framework for Learning Analytics. Journal of Educational Technology & Society.
4

10. Greller, W., & Drachsler, H. (to appear). Turning Learning into Numbers. Toward a Generic
Framework for Learning Analytics. Journal of Educational Technology & Society.
4

11. Stakeholders vs. LA Framework
Opinions from
the stakeholders
toward the
dimensions of the
LA framework
5

12. Learning Analytics Questionnaire
Extrapolate opinions from
different target groups:
(a) what is the current
understandings and the
expectations on LA
(b) is there a common
understanding of LA
6

13. Learning Analytics Questionnaire
• 4 weeks available
• 156 people after clean up
• 121 people full records
Data and survey available at:
http://bit.ly/la_survey
7

14. Participants
5.8%
11.0%
Higher Education
8.4% K-12
Vocational
Others
74.8%
8

15. Participants - Roles
%
50
44% Teachers
36% Researchers
37.5 L.Designers
26% Managers
25
16%
12.5
0
s s s s
er er er er
Te
ac
h
ar
ch sig
n
na
g Roles
es
e
L .D
e Ma
R
9

16. Participants - Reach
Responses from 31 countries [UK (38), US (30), NL (22)]
10

17. Stakeholders
data data
subjects clients
11

18. Stakeholders
(a) who was expected
to benefit the most
from learning analytics
Teachers
Parents
Institutions
Learners
12

19. Stakeholders
(a) who was expected
to benefit the most
from learning analytics
Outcomes:
1. Teachers
2. Learners
3. Institutions
4. Parents
Teachers
Parents
Institutions
Learners
12

20. Stakeholders
(b) how much will
learning analytics
influence bilateral
relationships?
Teachers
Parents
Institutions
13 Learners

21. Stakeholders
(b) how much will
learning analytics
influence bilateral
relationships?
Outcomes:
1. Teacher-student 84%
2. Student-teacher 63%
3. Student-student 46%
4. Teacher-teacher 41%
Teachers
Parents
Institutions
13 Learners

22. Objectives
Reflection
(Glahn, 2009)
14

23. Objectives
Reflection Prediction
(Glahn, 2009)
14

24. Objectives
The importance of 3 generic
objectives:
(a) reflection
(b) prediction
(c) unveil hidden information
15

25. Objectives
In which way learning analytics will change educational
practice in particular areas?
n = 119
11% no changes at all
43% small changes
45% extensive changes
16

26. Objectives
In which way learning analytics will change educational
practice in particular areas?
Item=2: Timely information
n 119
11% no changes at all
about learning
43% small changes
Item 8: extensive changes
45% Better insights by
institutions in their courses
Item 5: Easier grading
Item 6: Objective assessment
16

27. Educational Data
Drachsler, H., et al. (2010). Issues and Considerations regarding Sharable Data Sets for
Recommender Systems in Technology Enhanced Learning. 1st Workshop Recommnder
Systems in Technology Enhanced Learning (RecSysTEL@EC-TEL 2010) September, 28,
2010, Barcelona, Spain.
17

28. Educational Data
Drachsler, H., et al. (2010). Issues and Considerations regarding Sharable Data Sets for
Verbert, K., Manouselis, in Technology Enhanced Learning.E. (accepted).Recommnder
Recommender Systems N., Drachsler, H., and Duval, 1st Workshop Dataset-driven
Systems in Technology Enhanced Learning (RecSysTEL@EC-TEL 2010) September, 28,
Research to Support Learning and Knowledge Analytics. Journal of Educational Technology
2010, Barcelona, Spain.
& Society. 17

29. Educational Data
Researcher:
1. Added context information
(n=43, means 3.42)
Teacher:
1. Added context information
(n=52, means 3.42)
Manager:
1. Sharing within the institution
(n=16, means 3.63)
2. Anonymisation
(n=19, means 3.53)
18

30. Technologies
Manouselis, N., Drachsler, H., Verbert, K., and Duval, E. (to appear). Recommender
Systems for Learning. Berlin:Springer
19

31. Technologies
Prediction
Manouselis, N., Drachsler, H., Verbert, K., and Duval, E. (to appear). Recommender
Systems for Learning. Berlin:Springer
19

32. Technologies
Reflection
Peter Kraker, Claudia
Wagner, Fleur
Jeanquartier, Stefanie
N. Lindstaedt (2011):
On the Way to a
Science Intelligence:
Visualizing TEL Tweets
for Trend Detection
Sixth European
Conference on
Technology Enhanced
Learning (EC-TEL 2011)
Manouselis, N., Drachsler, H., Verbert, K., and Duval, E. (to appear). Recommender
Systems for Learning. Berlin:Springer
19

33. Technologies
Trust in accurate and appropriate LA results ...
1.View on learning progress
2. Predict learning resource
3. Assessment
4.View on engagement
5. Compare learners
6. Prediction of peers
7. Prediction of learner
performance
20

34. Constraints
1.Legal protection
2.Privacy
3.Ethics
4.Ownership
21

35. Constraints
Effect size of LA on ...
not at all a little very much
privacy ethics data data trans-
ownership openness parency

36. Competences
23

37. Competences
1.E-literacy
2.Interpretation skills
3.Self-directedness
4.Ethical understanding
23

38. Competences
24

39. Competences
Item 3: Critical reflection
Item 7: Self-directedness
Item 1: Numerical skills
Item 5: Ethical thinking
24

40. Competences
Item 3: Critical reflection
Item 7: Self-directednessthat
70.2% (n=85) believed
learners were NOT
competent enough to
Item 1: Numerical skills
independently learn from
learning analytics
Item 5: Ethical thinking
24

41. Limitations
picture by Stephan Downes http://www.flickr.com/photos/stephen_downes/3808714505
25

42. Limitations
1. Dominance of responses
from the HE sector
2. Absence of students
3. Low awareness of LA;
Only surveyed innovators and early adopters
4. Mainly opinions from western cultures
26

43. Survey summary
Main findings
of the survey
according to
the LA
framework
27

44. + Stakeholders
1.Main beneficiaries of LA are learners
and teachers followed by organisations
2.Biggest benefits would be gained in the
teacher-to-student relationship
3.Learners require teacher support to
learn from LA
28

45. + Stakeholders
1.Main beneficiaries of LA are learners
and teachers followed by organisations
2.Biggest benefits would be gained in the
teacher-to-student relationship
LA asto
3.Learners require teacher support
tio n?
learn from LA
inn ova
28

46. + Objectives
1. Reflection support is main objective from
the stakeholders view ...
2. ...by revealing hidden information about
learners
29

47. + Objectives
1. Reflection support is main objective from
the stakeholders view ...
2. ...by revealing hidden information about
learners
Reflection support only for
teacher student relationship?
29

48. + Educational Data
1. Context information from learners and the learning
process
2. Anonymisation is the second most important data
attribute
3. Willingness to share if data is anonymised
30

49. + Educational Data
1. Context information from learners and the learning
process
2. Anonymisation is the second most important data
attribute
3. Willingness to share if data is anonymised
Can we achieve a collection of
reference datasets?
30

50. + Technologies
1. Trust in LA algorithms is not well developed
2. High confidence on gaining a comprehensive
view of the learning progress
31

51. + Technologies
1. Trust in LA algorithms is not well developed
2. High confidence on gaining a comprehensive
view of the learning progress
How accurate can we measure
a learning progress?
31

52. + Constraints
1. Data ownership is the most important topic
2. LA lead to breaches of privacy but privacy and ethical aspects
are of lesser importance
3. Many organisations have ethical boards and guidelines in place
32

53. + Constraints
1. Data ownership is the most important topic
2. LA lead to breaches of privacy but privacy and ethical aspects
are of lesser importance
3. Many organisations have ethical boards and guidelines in place
Do we need new policies for
data ownership and privacy?
32

54. + Competences
1. Skepticism that LA will lead to more independence of
learners to control their learning process
2. Training need to guide students to more self-directedness
and critical reflection
33

55. + Competences
1. Skepticism that LA will lead to more independence of
learners to control their learning process
2. Training need to guide students to more self-directedness
and critical reflection
Do we need mandatory courses on
statistics for the edu. sector?
33

56. Future R&D
Future
R&D
picture by Tom Raftery http://www.flickr.com/photos/traftery/4773457853
34

57. 10 years of TEL RecSys research in one BOOK
Chapter 1: Background
Chapter 2: TEL context
Recommender
Chapter 3: Extended survey Systems for
of 42 RecSys Learning
Chapter 4: Challenges and
Outlook
Manouselis, N., Drachsler, H., Verbert, K., Duval, E.
(2012). Recommender Systems for Learning. Berlin:
Springer.
35

58. 10 years of TEL RecSys research in one BOOK
Chapter 1: Background
Chapter 2: TEL context
Recommender
Chapter 3: Extended survey Systems for
of 42 RecSys Learning
Chapter 4: Challenges and
Outlook
Manouselis, N., Drachsler, H., Verbert, K., Duval, E.
(2012). Recommender Systems for Learning. Berlin:
Springer.
35

59. Available TEL datasets
36

60. Recommender Technologies
37

61. Analysis according to the framework
Supported tasks
38

62. Analysis according to the framework
Supported tasks
Domain model
38

63. Analysis according to the framework
User model
39

64. Analysis according to the framework
Personalization Approach
40

65. Analysis according to the framework
Supported tasks
Domain model
User model
Personalization Approach
41

66. TEL RecSys::Ideal research design
1. A selection of datasets
for your RecSys task
2. An offline study of different
algorithms on the datasets
3. A comprehensive controlled user study
to test psychological, pedagogical
and technical aspects
4. Rollout of the RecSys in
real-life scenarios
42

67. TEL RecSys::Open issues
1. Evaluation
2. Datasets
3. Context
4. Visualization
5. Virtualization
6. Privacy
43

68. Addressing the issues::LinkedUP
LinkedUp
Web submissi
data on data(
Personal(
data
Stage(1J
Initialisation
Initialisation
36stages6of6the6LinkedUp competition6
LinkedUp Challenge6Environment
• Lowest(requirements(level(for(participation
• Inital(prototypes(and(mockups,(use(of(data( • LinkedUp Evaluation(Framework
Participation criteria
testbed(required • Methods and Test(Cases
Stage(2 • 10(to(20(projects(are(expected • LinkedUp Data(Testbed
• Competitor ranking list
• Medium(requirements(level(for(participation
• Working(prototypes,(minimum(amount of
data(sources,(clear(target(user(group
LinkedUp Support Actions
Stage(3 • 5(to(10(projects(are(expected( • Dissemination((events,( training)(
• Data(sharing(initiatives
• Deployment(in(realJworld( use(cases • Community(building(&(clustering
• Sustainable(technologies,(reaching out • Technology(transfer
to critical(amount(of(users,
Stage(4 • 3(to(5(projects(are(expected(
• Cashprice( awards(&(consulting
E
P S
T P F
Network(of(supporting(organisations( (
I
(see 3.2'Spreading'excellence,'exploiting'results,'disseminating'knowledge)'' S E
C B
C O
44

69. Addressing the issues::LinkedUP
LinkedUp
Web submissi
data on data(
Personal(
data
Stage(1J
Initialisation
Initialisation
36stages6of6the6LinkedUp competition6
LinkedUp Challenge6Environment
• Lowest(requirements(level(for(participation
• Inital(prototypes(and(mockups,(use(of(data( • LinkedUp Evaluation(Framework
Participation criteria
testbed(required • Methods and Test(Cases
Stage(2 • 10(to(20(projects(are(expected • LinkedUp Data(Testbed
• Competitor ranking list
the and
• Medium(requirements(level(for(participation
• Working(prototypes,(minimum(amount of
f LinkedUp Support Actions
t o ork os, al
data(sources,(clear(target(user(group
Stage(3
r • Dissemination((events,( training)(
pa etw
• 5(to(10(projects(are(expected(
r n • Data(sharing(initiatives
e n Eu atio ces
om Up
• Deployment(in(realJworld( use(cases • Community(building(&(clustering
0
• Sustainable(technologies,(reaching out
c d
Be ke 00 tern feren • Technology(transfer
to critical(amount(of(users,
Stage(4 • 3(to(5(projects(are(expected( 15 t in n • Cashprice( awards(&(consulting
n p to e a
Li u c oP
n c and E P S
Network(of(supporting(organisations( win nda ps T F
tte sho
I
(
(see 3.2'Spreading'excellence,'exploiting'results,'disseminating'knowledge)'' S C B E
a rk
44 w o C O

70. Addressing the issues::LinkedUP
45

71. !
Development*of*the*Evalua0on*Framework*
*
P1: Initialisation P2: Establishment P3: Exit and
and Evaluation Sustainability
M0-M6: Preparation M7-M18: Competition cycle M18-M24: Finalising
Comp
etition
Revie Final
Expert
3x
Draft w of
EF proposal EF release
validation of EF
New Refin
versio ement
n of EF
Literature review Group Concept Documentation
Cognitive Mapping Mapping Dissemination
Practical experiences
and refinement
Stefan Dietze 25/05/12 27
46

72. !
Group&Concept&Mapping&&
• Group Concept Mapping resembles the
Post-it notes problem solving technique
and Delphi method
• GCM involves participants in a few
simple activities (generating, sorting
and rating of ideas) that most people are
used to.
GCM is different in two substantial ways:
1. Robust analysis (MDS and HCA)
GCM takes up the original participants contribution and then quantitatively
aggregate it to show their collective view (as thematic clusters)
2. Visualisation
GCM presents the results from the analysis as conceptual maps and other graphical
representations (pattern matching and go-zones).
47 Stefan Dietze 25/05/12 1

73. !
Group&Concept&Mapping&&
Example: EU FP7
Handover
• 105 criteria about accurate handover training
interventions
• Sorting on similarity in meaning
• Rating on importance and feasibility
48 Stefan Dietze 25/05/12 31

74. !
Group&Concept&Mapping&&
A point map
49 Stefan Dietze 25/05/12 32

75. !
Group&Concept&Mapping&&
A cluster map
50 Stefan Dietze 25/05/12 33

76. !
Group&Concept&Mapping&&
Clusters’ labels
A
51 Stefan Dietze 25/05/12 34

77. !
Group&Concept&Mapping&&
Rating Map importance
52 Stefan Dietze 25/05/12 35

78. !
Group&Concept&Mapping&&
Rating Map feasibility
53 Stefan Dietze 25/05/12 36

79. !
Group&Concept&Mapping&&
54 Stefan Dietze 25/05/12 37

80. !
Practical experiences and refinement
Compe
titions
Review
Draft
3x
of EF
Refine
New
version
ment
of EF
55 Stefan Dietze 25/05/12 38

81. Many Thanks::Questions?
This presentation is available at:
slideshare.com/Drachsler
Email: hendrik.drachsler@ou.nl
Email: wolfgang.greller@ou.nl
Supporting projects:
!
56