Keynote #2, CITERS 2017 2:30 - 3:30 pm, 9 June 2017 (Friday) For detail please visit http://citers2017.cite.hku.hk/program-highlights/keynote-goldstein/

1. Incorporating digitally derived
information in large scale
educational assessment
programmes
Harvey Goldstein
University of Bristol

2. Large scale assessment
• Issues:
• Assessment for individual learning vs assessment for understanding
systems (‘accountability’) and educational processes: the latter will be
addressed.
• Information about attainment (tests, records) in relation to contextual
information such as SES, extra-institutional activities.
• Digital information may occur through tests themselves or through
contextual information e.g. derived from linked records either available
administratively or acquired through interrogating social media.
• A distinguishing feature of digital data is that it tends to raise new issues of
data quality, ethical issues and the volume of data to be processed.

3. Digitised (computerised) testing I
• Much assessment is now carried out using ‘adaptive’ algorithms that
present respondents with tasks based upon prior responses and/or
individual characteristics
• These typically depend strongly on a model for the relationship between
the response and one (possibly >1) underlying (latent) trait so that
information about the trait is continually updated to determine the next
task.
• The claimed advantage fro such a procedure is that it is ’efficient’ in terms
of time taken to reach a required accuracy on the trait, and that it is
implicitly ‘tailored’ so that the ‘difficulty’ of tasks is adjusted to match the
respondents ‘ability’.
• A major problem is the assumption of ‘unidimensionality’ and there is
anyway no guarantee that ‘convergence’ to the true trait value is assured.

4. Digitised (computerised) testing II
• More creative use of digitised testing is possible:
• Interactive test items can be utilised to elicit creativity
• Real life ‘project tasks’ can be simulated where projects are not feasible
• Feedback during testing can be utilised to ascertain responsiveness to such
feedback
• Choice of tasks can be implemented easily
• Large numbers of respondents can be used.
• Longitudinal information can be stored and built on.
• A formative assessment system can utilise the flexibility of a student
database.

5. Digitised contextual information. Acquisition
• Contextual information may be acquired either by direct
interrogation, e.g. via questionnaires or
• By linking respondents to existing databases
• Administrative data such as school records, health data, demographic family
data etc.
• Self generated data such as those on social media
• Issues:
• Ethical concerns about consent
• Data quality, especially from social media
• Vast quantities of potential data – how to select.

6. Digitally acquired information - quality
• If data sourced from admin records the quality of these may not be
apparent, nor the quality of the ‘linkage’ procedure.
• If sourced from social media, the context in which the data are
produced may be important, for example depending on whom the
perceived audience may be.
• This is not the case with traditionally collected data where the
environment is known and often strictly controlled such as in
experiments or interviews – there is research about the effect of data
collection environments on data quality and meaning.

7. International studies and digital data
• Studies such as PISA collect data largely in traditional ways within
fairly well controlled environments. Digital skills, such as reading
digital texts, navigating online etc have been incorporated.
• Is there a role for linking traditional forms of assessment to digital
media data (and other databases)?
• Would require cooperation with service providers such as facebook and
twitter
• Consent needed and safeguards against disclosure.
• Could provide interesting insights into students’ use of time and relationships
with traditional learning

8. Acquiring information digitally in general
• PISA already has plans for further assessing ‘digital literacy’ through
understanding internet ‘navigation’ and uses of online materials.
• Providing students with digital devices that monitor activities is a
further step and is already possible:
• Physical activities
• Learning activities
• Recording of life events and feelings
• Would seem to be worth developing:
• Can be controlled and quality assessed
• Can be motivating
• Can provide data unobtainable in other ways

9. Overview
• Large scale and especially international assessment systems cannot
ignore student use of digital information.
• A useful area for research is the use of ‘wearable’ devices to capture
dynamically student’s interactions with the real and virtual worlds.
• Devices to monitor physical activity are now common.
• A challenge exists to develop devices to monitor interactions with
others and to record different types of mental activity.
• A challenge also exists for data analysts to model such data.

10. Thank you for listening