2017-04-25 IEEE EDUCON in Athens - MOOQ Interactive Workshop on the Quality of MOOCs - Results from collaboration

1. MOOQ for the quality of MOOCs:
“We will make MOOCs better”
Quality Reference Framework with
indicators for design & comparison
www.MOOC-quality.eu
Frameworks: MOOQ

2. How to improve the
quality of Open
Education and MOOCs:
The Workshop Results
@ IEEE EDUCON 2017 in Athens
by Christian M. Stracke (OUNL)
and Cleo Sgouropoulou (NQIS)

3. Open CC License for
sharing & re-using slides
This work is free to share under the creative commons licence:
"Attribution – Noncommercial – Share Alike 3.0"
You can copy, distribute and transmit the work under the following conditions:
1. Attribution –
2. Noncommercial –
3. Share Alike
Licence: Attribution-Noncommercial-Share Alike
Some rights reserved, see: http://creativecommons.org/licenses/by-nc-sa/3.0/

4. Open University of the Netherlands
Global cooperation: ECNU & KNOU
Global initiative ICORE for OR & OE
Annual international LINQ Conference
eLC European Institute
Standardization Chair in ISO & CEN
Dr. Christian M. Stracke:
Open Learning & Education, Innovations,
Policies, Quality & Competences, Impact

5. Source: https://www.flickr.com/photos/opensourceway/

6. MOOQ for the quality of MOOCs:
“We will make MOOCs better”
Quality Reference Framework with
indicators for design & comparison
www.MOOC-quality.eu
Frameworks: MOOQ

7. Hands-on-Session
Explore the quality indicators of
MOOCs

8. Our proposal
for discussion:
The Quality Reference
Framework (QRF)

9. What is Open Education (OE)?
Quality dimensions of OE in practice?
Quality indicators for OE in practice?
Quality Open Education

10. 1. Pedagogical
2. Technological
3. Business Model
MOOQ Matrix

11. 1. MOOC Learners
2. MOOC Designers
3. MOOC Facilitators
4. MOOC Providers
MOOQ Target Groups

12. 16 questions in 4 clusters (n=45):
1. Experience with MOOCs (5 questions)
2. Perception and Expectation (5 questions)
3. Personal Goals (4 questions)
4. Recognition and Certification (2 questions)
n=45: reveals that many MOOC learners
do not share the intentions of the
MOOC designer & have their own goals
MOOQ Pre-Survey

13. Combines 3 surveys on 13 constructs:
1. MOOC learners (69 questions)
2. MOOC designers (89 questions)
3. MOOC facilitators (58 questions)
3 months with huge support (n=584):
reveals that most MOOC learners
reported positive learning experiences
with MOOCs
MOOQ Survey

14. Quality constructs
Constructs Learners Designers Facilitators
Pedagogical Decisions X
Learning Objectives X X X
Duration & Structure X X
Duration & Interaction X
Learning Resources X X X
Learning Support X X X
Flexibility & Inclusion X X
Learning Progress X
Learning Environment X X
Learning Assessment X X X
Learning Certification X X
Design Process X
Online Facilitation X

15. Stracke, C.M., 2016: “Opennessforlearning quality and change by Open Education in theory and practice - Overview,history, innovationsand policies:
How can Open Learning, OER and MOOCsachieve impact for learners, organizations and in society?” [in print]
ID ISO/IEC 19796-1 New ID Proposed Modification
NA Needs Analysis
AANN AAnnaallyyssiiss
FA Framework Analysis
CD Conception / Design DDEE DDeessiiggnn
DP Development / Production
PPRR PPrroodduuccttiioonn
IM Implementation
LP Learning Process LLEE LLeeaarrnniinngg
EO Evaluation/ Optimization
EEVV EEvvaalluuaattiioonn
OOPP OOppttiimmiizzaattiioonn

16. Stracke, C.M., 2016: “Opennessforlearning quality and change by Open Education in theory and practice - Overview,history, innovationsand policies:
How can Open Learning, OER and MOOCsachieve impact for learners, organizations and in society?” [in print]
Meso
Micro
Macro
Objectives Achievements
Realizations
AN
DE
PR
LE
OP
EV

17. OE Quality Dimensions
Evaluation
Analysis
Implemen-
tation
Learning
process
Design
Learner

18. QRF Target Groups
Meso
Micro
Macro
MOOC providers
MOOC designers
MOOC facilitators
MOOC learners
National ministries
Local authorities

19. How much do you agree with the
five phases?
(Analysis, Design, Implementation, Learning Process, Evaluation)
P-Q1: 5 Phases
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
0 1 5 7 5 1

20. The analysis phase is important
(Analysis, Design, Implementation, Learning Process, Evaluation)
P-Q2: Analysis Phase
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
0 0 0 1 17 0

21. The design phase is important
(Analysis, Design, Implementation, Learning Process, Evaluation)
P-Q3: Design Phase
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
0 0 0 6 13 0

22. The implementation phase is important
(Analysis, Design, Implementation, Learning Process, Evaluation)
P-Q4: Implementation
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
0 0 0 6 14 0

23. The learning process phase is important
(Analysis, Design, Implementation, Learning Process, Evaluation)
P-Q5: Learning Process
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
0 0 0 1 18 0

24. The evaluation phase is important
(Analysis, Design, Implementation, Learning Process, Evaluation)
P-Q6: Learning Process
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
0 0 1 4 14 0

25. How much do you agree with the
planned target groups?
(National ministries, local authorities, MOOC providers,
MOOC designers, MOOC facilitators, MOOC learners)
T-Q1: Target Groups
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
0 2 10 6 0 0

26. The national ministries are important
as target group
(National ministries, local authorities, MOOC providers,
MOOC designers, MOOC facilitators, MOOC learners)
T-Q2: National ministries
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
3 4 5 6 0 0

27. The local authorities are important
as target group
(National ministries, local authorities, MOOC providers,
MOOC designers, MOOC facilitators, MOOC learners)
T-Q3: Local authorities
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
2 3 7 5 1 0

28. The MOOC providers are important
as target group
(National ministries, local authorities, MOOC providers,
MOOC designers, MOOC facilitators, MOOC learners)
T-Q4: MOOC providers
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
0 0 2 7 8 0

29. The MOOC designers are important
as target group
(National ministries, local authorities, MOOC providers,
MOOC designers, MOOC facilitators, MOOC learners)
T-Q5: MOOC designers
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
0 0 1 6 11 0

30. The MOOC facilitators are important
as target group
(National ministries, local authorities, MOOC providers,
MOOC designers, MOOC facilitators, MOOC learners)
T-Q6: MOOC facilitators
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
0 0 1 9 7 0

31. The MOOC learners are important
as target group
(National ministries, local authorities, MOOC providers,
MOOC designers, MOOC facilitators, MOOC learners)
T-Q7: MOOC learners
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
0 1 2 8 7 0

32. How much do you agree that the
planned instruments are helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I-Q1: 9 Instruments
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
0 0 7 8 0 0

33. How much do you agree that the
planned MOOC is helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I-Q2: MOOC
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
0 1 5 8 2 0

34. How much do you agree that the
planned guidelines are helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I-Q3: Guidelines
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
1 0 3 8 2 0

35. How much do you agree that the
planned evaluation tools are helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I-Q4: Evaluation tools
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
0 1 4 7 3 0

36. How much do you agree that the
planned online communities are helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I-Q5: Online Communities
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
1 3 6 3 2 0

37. How much do you agree that the
planned design patterns are helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I-Q6: Design patterns
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
0 2 3 9 1 0

38. How much do you agree that the
planned checklists are helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I-Q7: Checklists
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
0 2 6 5 0 0

39. How much do you agree that the
planned use cases are helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I-Q8: Use cases
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
0 1 6 6 0 0

40. How much do you agree that the
planned authoring system is helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I-Q9: Authoring system
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
1 0 6 7 0 0

41. How much do you agree that the
planned OER collection is helpful?
(MOOC, Guidelines, Evaluation tools, Online communities,
Design patterns, Checklists, Use cases, Authoring system, OER)
I-Q10: OER collection
Fully
disagree: Disagree: Neutral: Agree:
Fully
agree: N/A:
0 0 4 5 4 0

42. Quality dimensions of OE in practice
Dimension 1: Analysis
Dimension 2: Design
Dimension 3: Implementation
Dimension 4: Learning Process
Dimension 5: Evaluation & Optimization
OE Quality Dimensions

43. Quality indicators for OE in practice
For Dimension 1:
For Dimension 2:
For Dimension n:
OE Quality Indicators

44. Quality indicators for analysis in OE:
• Budget
• Resources
• Learning outcomes
• Success indicators
• Adopted standardized methodologies
• Target audience
• Requirements for special needs
• Specification for all items above
OE D1: Analysis

45. Quality indicators for design in OE (1):
• Pedagogy
• Use of templates and story boards
• Simplicity and understanding
• Adaptability for different devices
• Design thinking: User experiences (user
interface design)
• Language
• Platform, medium and tools
OE D2: Design

46. Quality indicators for design in OE (2):
• Test and assessment at the end of each unit
• Interactivity
• Student communities
• Feedback questionnaire
• Traffic of the online platform
• Easy accessibility of the content
• Tracking of students‘ performance
OE D2: Design

47. Q. indicators for implement. in OE (1):
• Delivery modes
• Costs
• User authentification
• Teachers‘ background and skills
• Test plan
• Adaptivity for specific learner types
• Learning analytics
OE D3: Implementation

48. Q. indicators for implement. in OE (2):
• Feedback mechanisms such as gamification,
boni, and badges
• Existing standards such as ECTS
• Easy technical accessibility of the content
• Compliance with the learning standards
• Troubleshooting procedures
• Easy sharing of content and materials
OE D3: Implementation

49. Q. indicators for implement. in OE (3):
• Online user help
• Granularity
• Quick development processes
OE D3: Implementation

50. Quality indicators for learning in OE (1):
• Completion rate, Number of students
• Pre- and post tests of students‘ achievements
• Timing, Timeline and deadlines
• Participation in the forum
• Flexibility for students
• Feedback from questionnaires
• Efforts for the facilitators
OE D4: Learning Process

51. Quality indicators for learning in OE (2):
• Provided topics
• Interdisciplinarity and collaborations
OE D4: Learning Process

52. Quality indicators for evaluation in OE:
• Implementation and feedback to be used by
the developers
• Concept or resource-oriented evaluation
• Competency level of the students
• Goals and examination of the course
• Holistic evaluation system for whole MOOC
OE D5: Evaluation

53. Quality indicators for evaluation in OE (2)
• Delivery of learning outcomes according to
learning objectives
• Understanding of the added value of this
course compared to others
• Increase of the number of students
OE D5: Evaluation