The case for open learning analytics. Presented at LSAC 2018, Amsterdam, October 2018

1. The Case for
Open Learning
Analytics
Ian Dolphin
iandolphin2@mac.com
More about why this sign was created at http://www.intrastructures.net/Intrastructures/Actions_-_The_next_big_thing_2.html
PhotographIanDolphin|CCBY-SA4.0
Original Content CCBY-SA 4.0 International

2. Apereo Foundation Executive Director
EDUCAUSE Student Success Analytics
Community Group Steering Group

3. – The Apereo Foundation is a non-profit [501(c)(3)]
registered in New Jersey
Apereo Foundation Mission
“…collaborate to foster, develop, and sustain
open technologies and innovation to
support learning, teaching, and research."

4. A Personal Take

5. To become a convincing advocate for any technology,
you must become it’s sternest critic

6. Agenda
Learning Analytics - getting on the same page
Perceptions around Big Data
The Apereo Community approach
Jisc in the UK
Early lessons: Learning Analytics and Openness

7. Academic Analytics
System or organisation wide data
Learner/ing Analytics
Use data about individuals for action
Educational Data Sciences - the Broad Landscape …
Educational Data Mining
Use data to improve understanding of learning

8. Learning Analytics Definition
“Measurement, collection, analysis and
reporting of data about learners and their
contexts, for purposes of understanding
and optimizing learning and the
environments in which it occurs” (1)
(1) Learning and Academic Analytics, Siemens, G., 5 August 2011, http://www.learninganalytics.net/?p=131

9. Descriptive
Analytics
Diagnostic
Analytics
Predictive
Analytics
Prescriptive
Analytics
How can we
make it happen?
What will happen?
Why did it happen?
What happened?
Hindsight
Forsight
Insight
Information
Optimization
http://www.rosebt.com/uploads/8/1/8/1/8181762/6779091_orig.jpg, http://www.rosebt.com/blog/descriptive-diagnostic-predictive-prescriptive-analytics

10. Predictive Analytics
Applying techniques associated with big data to
data produced in course of learning.
Analysing historical aggregate data to identify
potential failure/success.
big data
noun COMPUTING
1. extremely large data sets that may be analysed computationally to reveal patterns,
trends, and associations, especially relating to human behaviour and interactions.

11. #LSAC17 Keynote @dgasevic
Recognising that learning analytics
is about learning - not just big data
#LearningAnalytics
Dragan Gasevic - Monash University and SoLAR

12. Controversy

13. The Guardian 5 March 2018 | https://www.theguardian.com/commentisfree/2018/mar/05/algorithms-rate-credit-scores-finances-data

14. Times Higher 7 Sept 2017 | https://www.timeshighereducation.com/features/will-learning-analytics-empower-or-entrap-students-and-academics

15. Hack Education Blog 11 Dec 2017 | http://hackeducation.com/2017/12/11/top-ed-tech-trends-weaponized-data

16. Choose your dystopia …

17. https://en.wikipedia.org/wiki/Ned_Ludd#/media/File:Luddite.jpg iLexx iStock photo ID:498260030 Upload date:December 02, 2015

18. Chaplin | Modern Times

19. If you’re involved in an aspect of
openness in education, the debate cannot
be ignored.
Open source communities should be aware of
the ethical dimensions of what they make.

20. Controversy - a Note
People write algorithms …
… or people write algorithms that write
algorithms …
It’s not difficult to find horror stories …

22. Wired 10 January 2015 | https://www.wired.com/2015/10/can-learn-epic-failure-google-flu-trends/
Google predicted where flu outbreaks would take place by what symptoms
are searched … but then changed their suggested search input to include “flu
symptoms” … 140% error on peak of 2013 flu season

23. LA Times 11 Oct 2016 | http://www.latimes.com/business/lazarus/la-fi-lazarus-big-data-healthcare-20161011-snap-story.html

24. The Guardian 30 June 2016 | https://www.theguardian.com/technology/2016/jun/30/tesla-autopilot-death-self-driving-car-elon-musk
Tesla algorithms failed to identify white truck against white sky

25. Wall Street Journal 1 July 2015 | https://blogs.wsj.com/digits/2015/07/01/google-mistakenly-tags-black-people-as-gorillas-showing-limits-of-algorithms/
Not intentional racism … but algorithms have unintended effects and consequences

26. (BTW, How’re you doing with that, Google …?)

27. The Verge 12 Jan 2018 | https://www.theverge.com/2018/1/12/16882408/google-racist-gorillas-photo-recognition-algorithm-ai

29. Photograph: Ian Dolphin | CCBY-SA 4.0

30. Nature 25 January 2015 | https://www.nature.com/news/reform-predictive-policing-1.21338

31. The Independent 11 May 2018 | https://www.independent.co.uk/news/uk/home-news/met-police-facial-recognition-success-south-wales-trial-home-office-false-positive-a8345036.html

32. The Independent 28 June 2018 | https://www.independent.co.uk/news/uk/home-news/facial-recognition-london-police-accuracy-human-rights-crime-database-a8422056.html

34. There are other approaches …
Hull Daily Mail 5 Oct 2013 | http://www.dailymail.co.uk/news/article-2445554/Hull-crook-caught-DNA-scene-took-bite-cucumber.html

35. Not a new controversy …

36. Ventriloquist - Own work
A scanned copy of a punched card given to me by a hydrologist around fifteen years ago.
• CC BY-SA 3.0
• File:Punched Card.jpg
• Created: 1 April 1997
Found https://en.wikipedia.org/wiki/Punched_card#/media/File:Punched_Card.jpg

37. Hollerith Card
“In 1936, it became mandatory for each Dutch
municipality to maintain a demographic record of
it’s inhabitants; by 1939, each citizen had to carry
a persoonkaart or personal identity card. Both
included a field for ‘Heritage’.”
“The Registry was maintained on punched
Hollerith cards.”
Greenfield, Adam: Radical Technologies pp60-61

38. 1940

39. http://web.annefrank.org/en/Subsites/Annes-Amsterdam/Timeline/Occupation/1943/1943/Attack-on-the-Public-
Records-Offices-on-the-Plantage-Kerklaan/#!/en/Subsites/Annes-Amsterdam/Timeline/Occupation/1943/1943/Attack-
on-the-Public-Records-Offices-on-the-Plantage-Kerklaan/

40. Wired 14 Dec 2017 | https://www.wired.com/story/age-of-social-credit/

41. EdSurge 9 April 2018 | https://www.edsurge.com/news/2018-04-09-china-s-netdragon-to-acquire-edmodo-for-137-5-million

42. Inside Higher Ed 21 Dec 2017 | https://www.insidehighered.com/news/2017/12/21/georgia-techs-monitoring-students-social-media-causes-concern
“…continuing to monitor this student’s social media accounts…”

43. Honi Soit 18 May 2017 | http://honisoit.com/2017/05/the-living-laboratory-how-the-university-watches-your-every-move/

44. Wonkhe 2 July 2018 | https://wonkhe.com/blogs/every-breath-you-take-new-moves-in-student-surveillance-and-support/
2nd July 18

45. The Telegraph 18th June 2018 | https://www.telegraph.co.uk/education/2018/06/18/university-students-data-shared-private-companies/

46. 18th June 2018
The Telegraph 18th June 2018 | https://www.telegraph.co.uk/education/2018/06/18/university-students-data-shared-private-companies/

47. Uninvent? Influence? Control?
The cat is out of the bag.
There is evidence of LA success.
Do we have the right to ignore this
evidence?
What safeguards are in place?
Some early lessons from UK, US, France

48. Apereo Community Activity
Based Around Open Source Code
A variety of small & large US institutions
Jisc
Pilots in France - including U Lorraine
Australian Universities

49. Learning Analytics Data Sources
Student Information Systems
Virtual Learning Environments
Attendance Monitoring Systems
Library Management Systems
Media Players
Learning Materials
??????????

50. Collect & Store
Intervention:
Action/Advising
Communication:
Dashboards/Early Alert
Analysis
Collect & Store
Intervention:
Action/Advising
Communication:
Dashboards/Early Alert
Big Data
Techniques
Analysis
Predictive Analytics

51. Spark
Hadoop
Collect & Store
Intervention:
Action/Advising
Communication:
Dashboards/Early Alert
Big Data
Techniques
Analysis
Communication:
Dashboards/Early Alert
Open Dashboard
Intervention:
Action/Advising
Student Success Plan
Big Data
Techniques
Collect & Store
Open LRW
Predictive Analytics
https://www.apereo.org/projects/shuhari
Uses historical aggregate data

52. Collect & Store
Intervention:
Action/Advising
Communication:
Dashboards/Early Alert
Analysis
Collect & Store
Intervention:
Action/Advising
Communication:
Dashboards/Early Alert
Rules
Based
Analysis
Rules Based
Doesn’t use historical aggregate data

53. Django
Pandas
ORM
Collect & Store
Intervention:
Action/Advising
Communication:
Dashboards/Early Alert
Rules
Based
Analysis
Communication
*and* Intervention:
OnTask Learning
Feedback at scale - via email -
framed as dialogue
Rules Based
https://www.ontasklearning.org/

54. Learning Analytics Data Sources
Student Information Systems
Virtual Learning Environments
Attendance Monitoring Systems
Library Management Systems
Media Players
Learning Materials
??????????
Moving from accidental to designed data sources

55. http://repository.jisc.ac.uk/6560/1/learning-analytics_and_student_success.pdf

56. Times Higher Education Supplement 28th June 2018 | https://www.timeshighereducation.com/news/uk-gets-world-first-national-learning-analytics-service

57. About Jisc
›UK’s national IT member organisation for
colleges and universities
›>600 member organisations
›Provide shared services, sector procurement
deals and support and advice
https://jisc.ac.uk
Thanks to Michael Webb, Jisc

58. Learning analytics use cases today
1) To support personal tutor system
2) To enable course leaders to improve courses
3) To enable students to understand their learning
Thanks to Michael Webb, Jisc

59. Supporting Personal Tutor System
Overview of the process:
1) Predictive analytics helps identify students at risk
2) Descriptive analytics helps identify why
3) Tutor determines intervention
Data Explorer
Thanks to Michael Webb, Jisc

60. Consent - Jisc View (1)
Most learning analytics is covered by:
a) legitimate interests of the organisation
b) necessary for the performance of a contract with the data
subject.
Consent isn’t the most appropriate basis for general learning
analytics.
There are exceptions.
See:
https://analytics.jiscinvolve.org/wp/2017/02/16/
consent-for-learning-analytics-some-practical-
guidance-for-institutions/
legal and contractual complexity
Thanks to Michael Webb, Jisc

61. Consent - Jisc View (2)
Consent is required for the following:
a) using sensitive characteristics
This includes attributes such as a person’s religion, ethnicity,
health, trade union membership or political beliefs.
This data may need to be used if using learning analytics to
support widening participation or addressing differential outcomes
b) for action based on automated decisions
This is explicitly covered under GDPR
See:
https://analytics.jiscinvolve.org/wp/2017/02/16/
consent-for-learning-analytics-some-practical-
guidance-for-institutions/
legal and contractual complexity
Thanks to Michael Webb, Jisc

62. Openness for predictive models
How do we explain our algorithms to
users in ways they can understand?
We want to avoid having a ‘Blackbox’
Non-technical staff can’t understand code.
understanding predictive models
Thanks to Michael Webb, Jisc

63. Institutions need to understand how the model works.
• Non data scientists need to understand the basic
approach, and be able to explain to students.
• Jisc are trying a guide for non-data scientists to
address this.
NB from ID - France, pilots intentionally include data
scientists
Openness for predictive models
understanding predictive models
Thanks to Michael Webb, Jisc

64. Openness as safeguard
Lessons of Early LA Work

65. Eight Dimensions of Openness
Open Purpose - be clear about the “why”
Open Ethical Framework -
Open and Inclusive Governance - learners, faculty
Open Source Software - a level of trust
Open Platform/Architecture - alternatives. It’s early.
Open Standards - ditto
Open Algorithms - more later
Open Consent/Consent Management (where required)
IanDolphinviaFlickrAttribution2.0Generic (CCBY-NC-ND2.0)

66. Challenges
Open Algorithms?
What do we need to effectively share?
A place/s to share?
Training data?
How do we increase understanding?
LAK18 Hackathon - “algorithmic literacy”
Curricular Impact?
GDPR and “right to understand”
NB: not a legal requirement
Potential role for counterfactuals?

67. ‘Counterfactual Explanations without
Opening the Black Box: Automated
Decisions and the GDPR’
Explains impact, enables recourse/subject action
“Counterfactual explanations take a similar form to
the statement: “You were denied a loan because
your annual income was £30,000. If your income had
been £45,000, you would have been offered a loan.”
Sandra Wachter, Brent Mittelstadt, & Chris Russell
https://arxiv.org/abs/1711.00399

68. Consent - Two Elements
Consent?
How well do we manage consent?
“Automated decision making” + sensitive info
What more do we need?
Mechanics of Managing Consent
(…and Managing withdrawal of consent)
Educating about purposes and rights
Enabling informed or educated consent

69. March 18th 2018
This is a perspective that is gaining ground
The Guardian 19 Mar 2018 | https://www.theguardian.com/commentisfree/2018/mar/19/the-guardian-view-on-data-protection-informed-consent-needed

70. Collectively, we can exert influence.
If we don’t …

71. ChrisRidellviaFacebook31March2018|https://www.facebook.com/authorchrisriddell/posts/1695060460546959