Every day, thousands of people comment on online news articles. While these discussions can encourage healthy debate, they can (and, especially in the absence of sufficiently engaged moderation, often do) also include insults, discrimination, or other harmful content. Since manually reviewing every comment is time-consuming and expensive, many news organizations and researchers are exploring how artificial intelligence (AI) can support content moderation.

In Felix’ master's thesis, which was carried out as part of the project, he investigated whether AI can make better moderation decisions when it has access to associated context instead of analyzing each comment in isolation. For example, is it helpful to add the news article's title or preceding comments as input? Furthermore, does it improve outcomes if we use Retrieval-Augmented Generation (RAG) to let the AI compare a new comment with similar, previously moderated examples before making a decision? To answer these questions, we evaluated several state-of-the-art large language models on a German-language dataset containing more than 11,000 comments that had been carefully annotated by professional moderators. We tested the models on different moderation tasks, including identifying inappropriate, discriminatory, and off-topic comments.

Our results show that context can improve AI moderation, but not in every situation. Larger generative language models benefited more consistently from additional contextual information, while smaller models sometimes became less accurate when too much information was provided. We observed a similar pattern for RAG: It improved performance for some moderation tasks but not for others. This highlights that there is no universal solution for AI-assisted content moderation. Instead, the most effective approach depends on the moderation task, the type of context, and the AI model being used.

The key message of Felix' master’s thesis is that better AI moderation is not simply about using larger models or adding more information. It is about understanding when context is helpful and designing moderation systems that are tailored to specific use cases. We hope these findings will contribute to the development of AI tools that effectively support human moderators and help create safer, more constructive online discussions. These findings are currently under submission in a peer-reviewed journal and we look forward to sharing the published work in the future.

References

https://www.derstandard.at/story/3000000261998/der-moderator-ein-tag-im-standard-forum

—

Assenmacher, D., Niemann, M., Müller, K., Seiler, M. V., Riehle, D. M., & Trautmann, H. RP-Mod & RP-Crowd: Moderator-and Crowd-Annotated German News Comment Datasets (Aug 2021). 

Yadav, A., Milde, B.: forumBERT: Topic adaptation and classification of contextualized forum comments in German. In: Proceedings of the 17th Conference on Natural Language Processing (KONVENS 2021). pp. 193–202. Düsseldorf, Germany (2021), https://aclanthology.org/2021.konvens-1.17/ 

—

Schabus, D., Skowron, M., Trapp, M.: One million posts: A data set of german online discussions. In: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. p. 1241–1244. SIGIR ’17, New York, NY, USA (2017). https://doi.org/10.1145/3077136.3080711