Trust & Safety Tools and Services

AbuseIOOpen source handling, ticketing and tracking for abuse reports.Case management · Open source
Aegis Content SafetyNVIDIA's dataset for content moderation and toxicity detection.Datasets and benchmarks · Open source
AILuminateHuman written prompts across harm categories, from MLCommons.Datasets and benchmarks · Open source
ALERTStandard and adversarial red teaming prompts, from Babelscape.Datasets and benchmarks · Open source
AltitudeFree desktop tool for reviewing and hash matching violent extremist content.Crisis and incident response · Open source
Aya Red-teamingMultilingual red teaming prompts across harm categories, from Cohere Labs.Datasets and benchmarks · Open source
AymaraAutomated evaluations for AI safety, accuracy and jailbreak vulnerability.Red teaming and evaluation · Open source
badwordsWord lists in several locales for flagging user submitted content.Datasets and benchmarks · Open source
Bluesky automodRules engine for automated moderation on AT Protocol services.Community management · Open source
CIB Mango TreeOpen tests for detecting coordinated inauthentic behavior.Investigations and OSINT · Open source
Content CredentialsOpen standard for recording where an image or video came from.Deepfake detection and provenance · Open source
Content Review FiltersReact components for putting blurs and filters into a review tool.Case management · Open source
CoopOpen source content review and enforcement: queues, routing, enforcement rules and reporting to NCMEC.Case management · Open source
CoPESmall open model that classifies content against a policy you define.AI content classifiers · Open source
CoralOpen source commenting platform with moderation built into it.Community management · Open source
CounterfitAutomation for assessing the security and robustness of AI models.Red teaming and evaluation · Open source
DarkBenchBenchmark for detecting dark design patterns in language models.Datasets and benchmarks · Open source
DeepEvalOpen source LLM evaluation with pytest style tests and a red teaming module.Red teaming and evaluation · Open source
DEF CON Red Teaming DatasetResults from DEF CON's generative AI red teaming event.Datasets and benchmarks · Open source
DetoxifyOpen models scoring text for toxicity, threats and identity attacks.Text and language analysis · Open source