„Making Sense of User Comments. Identifying Journalists’ Requirements for a Software Framework“, Vortrag von Wiebke Loosen, Marlo Häring, Zijad Kurtanović, Lisa Merten, Julius Reimer, Lies van Roessel und Walid Maalej (#SCAN-network: https://scan.informatik.uni-hamburg.de/) im Rahmen der Pre-Conference „Comments, Anyone? Multidisciplinary Approaches for Analyzing Online User Comments across News and Other Content Formats“ im Vorfeld der „67th Annual Conference“ der International Communication Association (ICA) am 25. Mai 2017 in San Diego, USA.

1. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Making Sense of User Comments
Identifying Journalists’ Requirements for a Software Framework
Wiebke Loosen, Marlo Häring, Zijad Kurtanović, Lisa Merten,
Julius Reimer, Lies van Roessel & Walid Maalej

2. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
User comments: an ambivalent
phenomenon
2
Shift in the understanding of user comments: from deliberative
hopes to being a “necessary evil”, or even a threat to deliberation
(e.g., Blom et al., 2014; Kropczynski et al., 2015; Pond, 2016)

3. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
The paradox of audience participation
▶ Acknowledging the need to include audiences while
trying to maintaining professional distance from
audiences (e.g., Heise et al., 2014; Schmidt et al., 2013; Singer et al., 2011;
WAN-IFRA & Huang, 2016)
▶ Many journalists engage with user comments
(e.g., Chen & Pain, 2016; Graham & Wright, 2015; Santana, 2011)
▶ Finding good user comments is like finding a needle in
a haystack! (cf. Braun & Gillespie, 2011)
▶ Handling user comments takes time, effort, and a
cross-platform approach
3

4. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
▶ Constructive approach: Help journalists (and others)
make sense of user comments with digital tools
▶ Formulate and validate requirements for a software
framework for analyzing user comments by means
of an iterative approach
1. Identify features and develop a first mock-up
2. Discuss mock-up with journalists and comment
moderators to refine requirements
Considerations & research goals
4

5. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Method

6. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017 6
Research design
Literature Review,
own preliminary
research
Explorative
Interviews
Mock-up
Development
Group Discussion
Editors
Group Discussion
Audience
Engagement Team
Mock-up &
Requirements

7. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Topics for group discussions
7
▶ Practices: handling user comments within the newsroom
▶ Challenges: pressing issues when dealing with user comments
▶ Quality: what makes comments especially valuable or helpful?
▶ Mock-up as ‘stimulus’ to discuss and develop possible software
features

8. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Results

9. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Group Discussion: General insights
▶ Technical assistance in ‘making sense of user comments’ appreciated
▶ ‘Multichannel function’ considered extremely useful
▶ Help is needed to identify ‘response-worthy’ comments
▶ Classifying pro- and contra-arguments particularly useful
▶ Visualizing dynamics of news discussions valuable for newsrooms and users
▶ Off-topic and redundant comments are annoying (particularly for moderators)
▶ Enabling ‘commenter typologies’ was welcomed, but also met with data protection
concerns
▶ Editors also regard user comments as constructive feedback more than moderators

10. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Analytical dimensions derived from
literature review & interviews
10
▶ Topics and actors
▶ Debate
▶ Quality indicators
▶ Comparison
▶ Sociodemographics
▶ Further features

11. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Mock-up: actors & topics
11
‘multichannel function’
regarded extremely
useful

12. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Mock-up: debate
12
Classifying pro- and contra-
arguments considered
particularly useful
Visualizing the dynamics of
news discussions regarded
valuable for newsrooms and
users

13. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Mock-up: quality indicators
13
Off-topic and redundant comments are
annoying (particularly for moderators)
Editors regard user comments as
constructive feedback and would like
response-worthy comments to be
filtered
‘commenter typologies’ welcomed, but
also caused data protection concerns

14. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Overview
of features
14

15. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Discussion

16. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
Challenges for technical implications
▶ Large volumes of typically natural, informal, and low-quality text across broad
range of topics
▶ Lexical and Syntactical machine learning needed to capture topics, actors &
arguments
▶ Availability of high-quality corpora of pre-labeled text
A lack of resources rather than a lack of ideas impede making sense of user
comments

17. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017 17
Thanks for listening!
https://scan.informatik.uni-hamburg.de/
Contact: w.loosen@hans-bredow-institut.de

18. ► Loosen et al. ┃ Making Sense of User Comments ┃ ICA San Diego ┃ May 25, 2017
References
▶ Blom, R., Carpenter, S., Bowe, B. J., & Lange, R. (2014). Frequent contributors within U.S. newspaper comment forums. An examination of their civility
and information value. American Behavioral Scientist, 58(10), 1314–1328.
▶ Braun, J., & Gillespie, T. (2011). Hosting the public discourse, hosting the public. When online news and social media converge. Journalism Practice,
5(4), 383–398.
▶ Chen, G. M., & Pain, P. (2016). Normalizing online comments. In: Journalism Practice. doi:10.1080/17512786.2016.1205954.
▶ Cision, & Canterbury Christ Church University. (2015). 2015 global social journalism study. Retrieved from http://cision-wp-
files.s3.amazonaws.com/de/wp-content/uploads/2015/09/Cision-SJS-2014-15-International.pdf
▶ Graham, T., & Wright, S. (2015). A tale of two stories from „below the line“. Comment fields at the Guardian. The International Journal of
Press/Politics, 20(3), 317–338.
▶ Heise, N., Loosen, W., Reimer, J., & Schmidt, J.-H. (2014). Including the audience. Comparing the attitudes and expectations of journalists and users
towards participation in German TV news journalism. Journalism Studies, 15(4), 411–430.
▶ Kropczynski, J., Cai, G., Carroll, J. M., Zhang, J., Puron-Cid, G., & Gil-Garcia, J. R. (2015). Characterizing democratic deliberation in an online forum.
Information Polity: The International Journal of Government & Democracy in the Information Age, 20(2/3), 151–165.
▶ Pond, P. (2016). Twitter time. A temporal analysis of tweet streams during televised political debate. Television & New Media, 17(2), 142–158.
▶ Rowe, I. (2015). Deliberation 2.0: comparing the deliberative quality of online news user comments across platforms. Journal of Broadcasting &
Electronic Media, 59(4), 539–555.
▶ Santana, A. (2011). Online readers’ comments represent new opinion pipeline. Newspaper Research Journal, 32(3), 66–81.
▶ Schmidt, J.-H., Loosen, W., Heise, N., & Reimer, J. (2013). Journalism and participatory practices – blurring or reinforcement of boundaries between
journalism and audiences? Recherches En Communication, (39), 91–109.
▶ World Association of Newspapers and News Publishers (WAN-IFRA), & Huang, C. L. (2016). Do comments matter? Global online commenting study
2016 (WPT/WEF Trends Report No. 1). Frankfurt/Main: World Association of Newspapers and News Publishers (WAN-IFRA).
18