A trial run that actually tests a moderation tool

Before a Trust & Safety team (the people who protect users from harm) buys a moderation tool, it usually runs a pilot: a few weeks of trying it out on real work. Most pilots don't actually test the tool. The seller drives, the examples are easy, nobody said what success looks like, and six weeks later the decision gets made on the warm glow of the demo and the pricing call.

A real test flips all of that. Your cases, your languages, your own reviewers doing the scoring, against goals written down before anyone shared a screen.

Before anyone demos anything

• Write down what success means first, with numbers, and get the team to agree. A goal invented after you see the results is just an excuse.
• Build your test set (more on that below) before the seller knows what's in it, and never share it. A test they've seen is a rehearsal.
• Include your least-used language and the kind of harmful content you find hardest. The seller's headline numbers were measured on neither.
• Decide who scores the results, how, and what happens when two scorers disagree.
• Set an end date, and the numbers that mean no. A pilot with no end date slowly turns into a subscription.

Building the test set

Collect a few hundred real examples where you already know the right answer, and load it up with the tricky ones. The obvious rule-breakers and the obviously fine posts check the basics. But you're paying for help with the messy middle, so that's where most of the examples go.

120  clear rule-breakers, across everything
     the tool says it can catch

120  clearly fine posts that look risky:
     news reports, people pushing back
     on hate, satire, medical and
     educational content

160  the messy middle: cases your own
     reviewers argued about this quarter,
     with the answer you settled on

  Every language you support, with extra
  weight on the smaller ones

What to measure

These are the numbers the sales deck didn't show you:

• When it flags something, is it right? This is called precision. Compare its calls to your known answers, for each kind of harmful content.
• Does it catch the tricky cases? This is called recall. Count how many of the 160 messy-middle cases it gets the way your team decided them.
• How it does in each language. The same two numbers, split by language, smallest market first.
• Reviewer time. Minutes per case in the new tool, compared with minutes in what you use now.
• What happens when it breaks. What your reviewers do during the seller's maintenance window. Ask, then watch.

The meeting afterwards

Score the tool against the goals you wrote before the demo, in writing, and keep the write-up. The next team to evaluate a tool will start from it, and it's your evidence when someone senior asks why you didn't pick the famous one.

A pilot that ends in a clear no has done its job. The expensive outcome is a so-so tool you adopted because six weeks of politeness piled up and nobody wanted to be rude. If the pilot can't fail, it isn't a pilot. It's a purchase with extra meetings.

Then share what you found. A proper review on the tool's page, with counts and context rather than a star rating, is exactly what the next team should be able to start from.