"Analysing User Comments in Online Journalism: a Systematic Literature Review", presentation at the International Communication Association's (ICA) 69th Annual Conference on May 28th, 2019 in Washington, D.C. (together with Volodymyr Biryuk, Marlo Haering, Wiebke Loosen, Walid Maalej and Lisa Merten).

1. ANALYSING USER COMMENTS IN
ONLINE JOURNALISM
A Systematic Literature Review across
Communication Studies and Computer Science
69th annual conference of the International Communication Association (ICA)
J. Reimer, V. Biryuk,
M. Haering, W. Loosen,
W. Maalej, & L. Merten
28 May 2019
Photo:SebastianSiggerud,Unsplash

2. @LisaMerten
@marlohaering
@julius_reimer
@Wloosen
@maalejw
@VolodymyrBiryuk
Photo:SebastianSiggerud,Unsplash

3. COMMENT ANALYSES ARE A WIDE FIELD –
HOW GET AN OVERVIEW?
Photo:SebastianSiggerud,Unsplash

4. ANALYSING USER COMMENTS IN ONLINE JOURNALISM | REIMER ET AL. | 4
AIM OF THE STUDY
§ Overview of content analyses of comments
on journalistic stories with respect to
§ what is studied (aspects of comments)
§ how it is studied (manual qualitative,
manual quantitative, (semi-)automated
content analysis)
§ in what discipline (communication studies,
computer science, other)
§ Determine under-researched aspects
§ Identify potential for interdisciplinary
collaboration
Photo:AnnieSpratt,Unsplash

5. ANALYSING USER COMMENTS IN ONLINE JOURNALISM | REIMER ET AL. | 8
METHOD
§ Systematic literature review of
§ content analyses
§ of user comments
§ that refer to journalistic stories,
§ published before 2017.
Additional searches:
• SC|M
• Medien & Kommunikationsw.
• Coral Project reading list
• Springer Link
• Project literature database
• ScienceDirect
• Web of Science
• Google Scholar
Repositories searched:
• EBSCO CMMC
• ACM Digital Library
• IEEE Explore
Search string representing inclusion criteria
2.220 potentially relevant studies
Examination of title, keywords, abstract (+ full text)
203 relevant studies

6. ANALYSING USER COMMENTS IN ONLINE JOURNALISM | REIMER ET AL. | 9
CODEBOOK
Variable Example (sub-)categories
Bibliographical information Authors, publication year, discipline, etc.
Methodology
Comment analysis method applied, additional methods applied,
features/algorithms used in automated approaches, reliability/evaluation
scores, etc.
Sampling
Media outlets & news stories comments refer to, number of analysed
comments, etc.
Construct categories/
variables measured
Quantitative aspects Length of comments, number of comments per story, etc.
Kinds of content
Personal opinion/attitude, argument for opinion, additional
information/material, media criticism, etc.
Incivility Offensive language, personal insults, racism, sexism, etc.
Addressees Other users, journalist, forum moderator, etc.
Emotionality Anger, hatred, fear, surprise, humour, etc.
Readability Sentence length, technical/foreign terms, etc.
Facticity Correctness of facts stated in comments
Other variable/construct (Open category)

7. ANALYSING USER COMMENTS IN ONLINE JOURNALISM | REIMER ET AL. | 10
(INSTEAD OF) RESULTS: A JOINT AGENDA FOR FUTURE RESEARCH
Photo:JESHOOTS.COM,Unsplash

8. ANALYSING USER COMMENTS IN ONLINE JOURNALISM | REIMER ET AL. | 13
LOOK AT THE GLOBAL SOUTH;
TV, RADIO, DIGITAL NATIVES, TABLOIDS;
LESS WIDELY SPOKEN LANGUAGES
§ Nearly 50 % of studies on UK/US
comments, while Global South –
especially Africa – is widely
disregarded
§ Strong tendency towards
broadsheet newspapers
§ Automated analyses focus on
widely spoken languages
Photo:AndrewStutesman,Unsplash

9. ANALYSING USER COMMENTS IN ONLINE JOURNALISM | REIMER ET AL. | 16
LOOK AT
‘CONSTRUCTIVE’ COMMENTS,
PROPAGANDA, & FACTICITY
§ Analyses, particularly automated ones,
seldomly concerned with positive or
useful aspects of comments
§ Only one study deals with propaganda
in comments
§ Hardly any fact-checking of users’
contributions
Photos:geralt/GerdAltmann,Pixabay
EhimetalorUnuabona,Unsplash
Emanuele,flickr

10. ANALYSING USER COMMENTS IN ONLINE JOURNALISM | REIMER ET AL. | 17
LOOK AT
FACEBOOK, TWITTER, YOUTUBE,
EVEN DARK SOCIAL
§ 90 % of studies are concerned
with comments on news websites
themselves
§ Comments on social media rarely
analysed (Facebook: 9 %, Twitter:
5 %, YouTube: 5 %)
§ Only 9 % compare comments
from different platforms

11. ANALYSING USER COMMENTS IN ONLINE JOURNALISM | REIMER ET AL. | 19
MORE INTERDISCIPLINARY
COLLABORATION & TRANSFER OF
KNOWLEDGE
§ Shared interests: nearly all aspects
investigated in both communication
studies & computer science
§ But: only 3 of 454 authors published in
both disciplines
Photos: Mimi Thian, Unsplash

12. ANALYSING USER COMMENTS IN ONLINE JOURNALISM | REIMER ET AL. | 20
HELP COMPUTER SCIENTISTS
DEVELOP AUTOMATED APPROACHES
§ Computational analyses gain
importance in communication studies
§ But tools predominantly developed by
computer scientists
§ Comm. Scholars can provide:
§ Phenomena of interest & theories to
develop RQs
§ Field expertise for operationalisation
& interpretation of results
§ Qualitative ‘pre-studies’ (e.g.,
addressees, different forms of hate
speech, ‘constructive’ comments)
Photo:FranckV.,Unsplash

13. ANALYSING USER COMMENTS IN ONLINE JOURNALISM | REIMER ET AL. | 21
CONTRIBUTE TO MULTI-METHOD
APPROACHES
§ Multi-method studies only common in
communication studies
§ Computer science mostly looks at
comments in isolation
§ Interviews, surveys, analyses of
commented articles, etc. can add
context & help improve automated
approaches
Clipart:AnnetteSpithoven,nounproject
NikkiRodriguez,nounproject
Rflor,nounproject

14. ANALYSING USER COMMENTS IN ONLINE JOURNALISM | REIMER ET AL. | 22
PRODUCE BETTER TRAINING DATA
TO IMPROVE MACHINE LEARNING
§ Manually coded training data is
essential for automated approaches
based on supervised machine learning
§ But: lack of transparency (& of rigour?)
regarding theoretical foundation,
operationalisation, reliability,
qualification of coders
§ Communication scholars could provide
expertise in manual content analysis
§ Better training à better performance
Photo: Sven Mieke, Unsplash

15. ANALYSING USER COMMENTS IN ONLINE JOURNALISM | REIMER ET AL. | 23
TEAM UP FOR MULTI-STEP ANALYSES
§ Different epistemological interests, or:
degrees of complexity reduction:
§ detection of certain aspects in comments
§ vs. analysis of the very nature of aspects
§ Promising automated approaches for
§ detection of trolling/spam, ‘hot topics’,
exceptional statements, off-topic
comments
§ determination of sentiments, discussion
structure, diversity
Automatic detection &
sampling of comments of
interest
Manual in-depth analyses

16. ANALYSING USER COMMENTS IN ONLINE JOURNALISM | REIMER ET AL. | 24
LIMITATIONS
§ All relevant studies included?
§ What happened 2017–2019?
§ (Predominantly) quantitative
analysis, i.e. high degree of
complexity reduction
§ Interview, survey, & experimental
studies not included
Photo: Oscar Sutton, Unsplash

17. Julius Reimer
Volodymyr Biryuk
Marlo Haering
Wiebke Loosen
Walid Maalej
Lisa Merten
THANK YOU
Photo:ScottVanHoy,Unsplash