Every large online forum faces the same challenge: thousands of comments a day, a handful of moderators, and tools that hand back a single number. 0.83 toxic. Act on it – or don't. When a commenter appeals, "the model said so" is not a very satisfying answer.

HaSPI is our attempt at providing the missing link: a scoring method for German forum comments that reports which words drove the verdict – exactly, not approximately. In this (final) blog post we want to summarise what happened in the past months and give you the final results.

A Lesson from Robotics

The underlying method comes from inverse reinforcement learning, a type of imitation learning. The central idea behind this is to watch an expert perform a task and infer the reward function they must have been optimising for. We apply the same principle to moderation: Given the comments that moderators allowed and the comments that they removed, what implicit reward function separates them?

Drawing upon a comprehensive literature research on the current methodological state of the art, we derived a theoretical framework for the classification from a reward (which you can read about in our previous blogpost). This was followed by months of implementation, testing and iterating different designs.

Almost all of our early attempts failed. While the inherent explainability of our method worked great, the classification results were sub-par at best – and no better than random chance at worst. Then, suddenly, we got results that were essentially state of the art! 

What changed? As it turns out, the underlying encoder (i.e. what is used to “understand” the underlying text) is really important (duh!). We spent a lot of time verifying our method using small and fast, but also very outdated encoders – switching to a modern, state-of-the-art transformer elevated the performance!

A Bit on Context

Context is really important for moderation decisions, as the same comment can be completely harmless under one article but clear hate speech under another. We therefore incorporated optional context into our tool. As shown in the example below, our model can indeed distinguish based on context in typical context-dependent examples.

When we quantitatively evaluated the effect of context it appeared to make a huge difference.

However, splitting the data at random, the improvements apparently came from the model memorising which articles especially attract moderation. Splitting the data so no article appeared in both the train and the test set prevented leakage but also reduced model performance. We flag this because published context-aware moderation results often do not clarify whether or not they used random or article-disjoint splits, so some other methods’ results are likely inflated the same way.

Results

The key takeaway: On the RP-Mod benchmark, our approach matches a fully fine-tuned German BERT, even with a frozen encoder training no weights, while simultaneously staying interpretable. Two independent methods hitting the same performance ceiling may indicate that this is the maximum predictability that can be expected from comment text alone.

Try It Yourself

pip install "haspi[leolm,demo]"
haspi-explain # type a comment, get a verdict and the words behind it
haspi-demo    # web app: risk meter plus colour-coded contributions

In our opinion, aside from the presentable results, the key finding is that you do not need to sacrifice model performance for interpretability.

HaSPI is open source, built on the One Million Posts Corpus (derStandard.at), aimed at moderators who want to test it on their own material and researchers who want to  reproduce or extend it. Check out the repository and documentation!