Trust & Safety Tools and Services

Aegis Content SafetyNVIDIA's dataset for content moderation and toxicity detection.Datasets and benchmarks · Open source
CoPESmall open model that classifies content against a policy you define.AI content classifiers · Open source
DetoxifyOpen models scoring text for toxicity, threats and identity attacks.Text and language analysis · Open source
DiscourseOpen source forum platform with trust levels, flagging and moderation built in.Community management · Open source
gpt-oss-safeguardOpen weight reasoning model that classifies text against a policy you write.AI content classifiers · Open source
Granite GuardianOpen guardrail models for detecting harms in LLM traffic.AI guardrails · Open source
HateBERTBERT retrained on banned Reddit communities for abusive language detection.Text and language analysis · Open source
Jigsaw ToxicityWikipedia comments labeled by human raters for toxic behavior.Datasets and benchmarks · Open source
Lemmy moderationModeration tools built into the open source federated link aggregator.Community management · Open source
Llama GuardOpen weight model that checks prompts and responses against a written policy.AI guardrails · Open source
Mastodon moderationReports, account actions, filters and domain blocks built into Mastodon.Community management · Open source
OSmodOpen source moderation tooling and APIs from Jigsaw.Content moderation platforms · Open source
OzoneOpen source labeling and moderation console for the AT Protocol.Case management · Open source
Roblox Voice Safety ClassifierOpen model flagging harmful speech in live voice chat.AI content classifiers · Open source
ShieldGemmaOpen weight safety classifiers for model inputs and outputs.AI guardrails · Open source
ToxiGenLarge machine generated dataset of implicit hate speech across 13 minority groups.Datasets and benchmarks · Open source
WildGuardOpen moderation model for prompts and responses, with a benchmark of real and adversarial cases.AI guardrails · Open source