Build or buy your moderation tools

Trust & Safety teams, the people who protect users from harm, run on software: tools that spot trouble, track reports and help reviewers make the call. Every team starts with a spreadsheet. Every team outgrows it. And then somebody asks whether to build the real tools or buy them.

There's no single answer, and anyone who gives you one is selling something. It depends on the tool, and on two questions: how much of your problem is unique to your platform, and how bad it is when the tool gets it wrong.

Tool by tool

The usual answer for each piece, and why, goes like this. Your team may be the exception, but check twice before deciding it is.

• Detecting child sexual abuse material: buy, and join the shared industry lists. Companies share digital fingerprints of known material so everyone can catch it, and holding this material to train your own system is a legal problem. Never build this alone.
• Automatic detection (software that flags content that might break the rules): buy, then adjust. Vendors (the companies selling it) split the cost of training it across many customers. Your job is testing how well it catches your problems, in your languages.
• Case management (where reports land and get worked): buy early, and look again once you're big. A list of cases, a record of who did what, and reports all come off the shelf, until your way of working stops fitting the shelf.
• Report and appeal buttons for users: build. They live inside your product, and how they work is a decision about your rules that no vendor can make for you.
• Transparency reports (public numbers on what you removed): it depends. Buy the formats regulators ask for, and build the pipes that feed them. The data is yours either way.
• Reviewer wellbeing features: buy, and insist on them. Blur, black-and-white and mute controls belong inside whatever review tool you pick. Make them a requirement when you buy, not a project for later.

Costs that aren't in the quote

• Connecting it to your systems is the real project. The license has a price. The eighteen months of plumbing and odd cases doesn't.
• How well it works in each language. That big accuracy number was measured somewhere, on something, and probably not in your smallest market. Ask for results per language, on your kinds of problems.
• Your history ends up in their format. Before you sign, get in writing what leaving looks like: the export format, the cost and how long it takes.
• Price it at next year's volume. Per-person or per-item pricing that feels fine today is a very different number after a good growth quarter.
• What they do when it breaks. Their outage means your reports pile up, so ask what happened during their last one.

When building is the right call

Building wins when your problems or your product really are unusual, when engineering time is what you actually have plenty of, or when you're big enough that paying per item stops making sense. Those are all real reasons, and the platforms that build well build for them. They also count the cost of the second job honestly.

That second job starts the day the tool ships. From then on, you own:

• Being on call. It breaks at 2am, and someone's phone rings.
• The wishlist. Every rule change is now a ticket for engineers.
• The screens. Reviewers use it 8 hours a day, and nobody was hired to design it.
• Moving off it. Leaving your own tool hurts as much as leaving a vendor's, and you can't even blame the vendor.

None of that says don't build. It says decide with the running costs in front of you. The build-or-buy meeting always prices the building and almost never prices the owning. A tool you built is a product you now run, so price the second job before you take the first one.

Doing your homework

If you're buying, start with the Tools & Services catalog: real products, what each one costs in plain words, and reviews that say whether a team would use it again, shown as counts rather than stars. Every category page has six questions worth taking into a sales call, and the compare view lines up two or three tools side by side.

The best research is still asking a team your size what they use and what they regret. It's a normal question in this field, and people answer it honestly. Sometimes too honestly.