It’s not news that the comments sections of online news sites can be hot spots for sharing ill-informed views, ad hominem attacks, or just good old fashioned vituperation. A recent study out of Germany finds that online comments – even polite, well-reasoned ones – can also hurt the perceived quality of news stories.
One reason this is worth noting is that people in the U.S. get a lot of their science news from online sources – and that number is growing. A couple years ago, I wrote about a National Science Foundation report that found 42 percent of folks in the U.S. got their science news online in 2012. And that number jumped to 47 percent by 2014, according to a report released by the American Association for the Advancement of Science (AAAS) last month.
Learning that comments affect the way readers interact with information isn’t necessarily a big surprise. A study I wrote about in 2013 reported that reader comments on online news stories actually influence how readers view the subject of the story – in that case, uncivil comments influenced readers’ risk perceptions about nanotechnology. (You can find the 2013 study itself here.)
But the recent German study appears to be the first to look at whether online comments influenced how readers perceive the actual journalistic quality of the news story, as well as whether the civility and reasoning of those comments made a difference. Further, the study looked at whether the “brand,” or reputation, of the news source made a difference. Short version: comments of any kind hurt the perceived quality of the news story itself. But, of course, it’s more complicated than that. My synopsis is below, but I encourage you to read the study – it’s clearly written and includes far more detail than I’ll go into here.
The study, “Effects of civility and reasoning in user comments on perceived journalistic quality,” was published online March 22 in the journal Journalism Studies. The paper was co-authored by Fabian Prochazka, Patrick Weber, and Wolfgang Schweiger.
The study addressed two research questions:
- “How do civility and reasoning in comments affect perceptions of journalistic quality in known versus unknown news brands?”; and
- “How is the journalistic quality of a news item accompanied by user comments of varying degrees of civility and reasoning judged in comparison to the same article without accompanying comments?”
The researchers define “reasoned” comments as those in which commenters provide evidence for their statements. Civil comments are defined as those that are conveyed in a “polite, respectful manner.”
For the study, researchers recruited 942 people (60.1 percent women, average age 32.2) to participate in an online survey.
Each study participant was shown one of 15(!) different variations of a news story that compared risks related to marijuana and alcohol use. The story could be linked to any of three news brands; two (Spiegel Online and Focus Online) are well-established German news sites, and one was a fictitious news site (Fakt Aktuell) that the researchers made up to ensure that study participants had never heard of it. For each news brand there were four combinations of civil versus uncivil comments and reasoned versus unreasoned comments, as well as a control version that included no comments.
For each version of a story with comments, there would be two comments in favor of legalizing marijuana and two comments opposed to legalization.
The study participants were then asked seven questions that addressed six aspects of journalistic quality: accuracy; impartiality; relevance (i.e., relevance to society); diversity (i.e., incorporated different points of view); ethics; and comprehensibility. There were seven questions because there were two questions related to ethics. The questions addressed each of these issues using a five-point scale. For example, study participants were shown the statement “The article is objective,” and asked to rate it between 1 (“I do not agree at all”) to 5 (“agree completely”).
The researchers had to toss “accuracy” out, because of statistical stuff that I don’t understand. The remaining five aspects of journalistic quality ended up sorted into two groups.
One group was called “informational quality,” and included impartiality, relevance, and diversity. The second group was “formal quality,” and included ethics and comprehensibility.
The civility and reasoning of comments had no impact on how participants viewed informational quality, with one exception. The unknown news brand (Fakt Aktuell) was rated lower when accompanied by unreasoned comments compared to when it was accompanied by reasoned comments. However, even here, the effect was only of marginal statistical significance.
Brand also played a role in informational quality when looking at the control group – which included no comments. When study participants were shown the story attributed to the unknown news brand, unreasoned comments fared worse on informational quality than the control group. However, the control group fared the same as the group that had reasoned comments. In other words, as the authors note, “For the unknown brand….judgments of an article’s informational quality were not positively biased by high-quality (i.e., reasoned) comments, but were negatively biased by unreasoned comments.”
Known brands fared worse. The researchers found that “informational quality ratings in the no-comments control condition were higher than in both the reasoned-comments condition and the unreasoned-comments condition.” In other words, the authors write, “an article was rated as being lower in quality merely due to the presence of comments.”
Brand had less effect on formal quality. Study participants rated formal quality lower when accompanied by uncivil comments compared to civil comments, regardless of brand. That doesn’t mean that civil comments boosted perceptions of formal quality, but that uncivil comments hurt. It didn’t matter whether the comments were reasoned.
And the control group outperformed any type of comments. As the paper states: “the mere presence of comments deteriorates perceptions of formal quality.”
Let’s get several qualifiers out of the way. First, this study was done in Germany, not the U.S., so it’s not clear how applicable this is to U.S. audiences. (If anyone is repeating this experiment in the U.S., I’d love to hear about it.) Second, the effects found in the study are not pronounced. Lastly, it’s not clear whether this finding would extend to other online stories that aren’t on conventional news media sites. For example, would we see a similar impact from comments on a news site maintained by a university or other research organization? I don’t know.
But the findings are still troubling.
As I noted at the top, many people in the U.S. get their science news online. If science writers and communicators are hoping to engage with audiences online, they presumably want to be taken seriously. No one wants the quality of their work to seem worse simply because there’s a comment box at the bottom of the page.
In 2013, I wrote that I was – on balance – in favor of allowing comments: “When in doubt, my default is always to encourage dialogue (even though it can get ugly). I’ll certainly always allow comments on this blog – though I’ll delete those I deem off-topic or offensive. But many online news outlets don’t have the resources to police their comments, or the will to weigh in – especially if it might mean alienating a segment of their readership.”
I’d want to see more research on this issue – a lot more – before reaching any firm conclusions, but I’m beginning to think Popular Science may have had the right idea when it eliminated online comments back in 2013.
In the meantime, I’m not getting rid of comments here – so please weigh in with your thoughts on this (preferably in a civil, reasoned way).