A lot of Trust & Safety work happens in a queue: a line of reported posts and accounts waiting for a person to look at them. The whole operation gets measured with about five numbers, and you can work a queue for years without anyone explaining what they mean.
Then an interviewer asks about "precision", or your yearly review reads your quality score back to you, and suddenly you need the words. Good news: there aren't many of them.
The five numbers
- Volume: how much comes in, and from where. Careful, though. The number of reports tracks how much people feel like reporting, not how much harm there is, and one news story can move it.
- Precision: of everything you took action on, how much really broke the rules. It means nothing unless you know what sample it came from.
- Recall: of everything that broke the rules, how much you caught. To know it you need to know what you missed, and the queue can't tell you that.
- Prevalence: how much rule-breaking content is out there at all. It's measured by checking a random sample of everything, not by counting reports.
- Response time: how fast cases get handled, with a separate target for each kind of problem. An average hides the slowest cases, and the slow ones are where the damage happens.
Precision and recall, with real-looking math
One more term first. "QA" (quality assurance) means a second reviewer rechecks a sample of decisions to see if they were right. Here's how it plays out for one rule over one week:
Took action on this week 400 posts QA rechecks a sample 100 of them QA agrees with 91 Precision ~91% Separately: a random sample of posts nobody reported turns up rule-breakers the queue never saw at all. Est. rule-breakers, total ~600 this week Caught 400 x 0.91 = ~364 Recall ~61%
Made-up numbers, real shape. The two pull against each other. Make the rule stricter and precision goes up while recall goes down. Loosen it and the reverse happens. A team told to raise both at once is being asked to beat a trade-off, and the honest answer says what the trade costs.
Notice where that 600 came from. Prevalence comes from checking what nobody reported. The queue only ever shows you what somebody did.
The eight-minute case
Most reviewers get scored on how many cases they finish and on their quality score, both at once. The two collide on every hard case. The decision that takes eight minutes to get right is exactly the one the hourly target punishes.
Which number wins in a tie is a question for the team's rules. On a well-run team the answer is written down, so nobody has to guess under pressure. If nobody has written it down where you work, asking for it in writing is a quietly senior move.
Using the numbers well
- When someone shows you a number, work out which of the five it really is. Plenty of dashboards say "accuracy" and mean something else entirely.
- Never quote a precision figure without its sample. "91% on a QA sample of 100 cases" is information. "91%" is decoration.
- When a target changes, ask what it costs. A faster response time gets paid for somewhere, usually in quality.
- Bring one number of your own to your yearly review: how many cases you handled, how your quality score moved, a delay you fixed. Good resume lines start here, in the week the numbers happen.
- Be suspicious of a number that never moves. Real measurements wobble. A perfectly flat line is usually a spreadsheet nobody has opened since March.
Further reading
- AI content classifier tools. Software that flags bad content automatically, sold using these same numbers