Trust & Safety Tools and Services

Aegis Content SafetyNVIDIA's dataset for content moderation and toxicity detection.Datasets and benchmarks · Open source
AILuminateHuman written prompts across harm categories, from MLCommons.Datasets and benchmarks · Open source
ALERTStandard and adversarial red teaming prompts, from Babelscape.Datasets and benchmarks · Open source
Aya Red-teamingMultilingual red teaming prompts across harm categories, from Cohere Labs.Datasets and benchmarks · Open source
badwordsWord lists in several locales for flagging user submitted content.Datasets and benchmarks · Open source
DarkBenchBenchmark for detecting dark design patterns in language models.Datasets and benchmarks · Open source
DEF CON Red Teaming DatasetResults from DEF CON's generative AI red teaming event.Datasets and benchmarks · Open source
Do Not AnswerQuestions across risk areas for testing safety and refusal behavior.Datasets and benchmarks · Open source
DSA Transparency DatabaseEvery statement of reasons filed by an EU platform, downloadable daily.Audit and reporting · Free
GIFCT Hash-Sharing DatabaseShared hashes of terrorist and violent extremist content, for members.Hash matching and databases · Free to qualifying organizations
HackAPrompt DatasetPrompt based jailbreak attempts, for testing model vulnerability.Datasets and benchmarks · Open source
HarmBenchEvaluation dataset for automated red teaming, from the Center for AI Safety.Datasets and benchmarks · Open source
IWF Hash ListAnalyst verified hashes of child sexual abuse imagery, updated daily.CSAM detection · Subscription
JailbreakBenchHarmful behaviors used as a benchmark for jailbreaking evaluation.Datasets and benchmarks · Open source
Jigsaw ToxicityWikipedia comments labeled by human raters for toxic behavior.Datasets and benchmarks · Open source
LanternCross platform signal sharing for child safety violations.Hash matching and databases · Free to qualifying organizations
LumenArchive of the takedown and removal requests sent to online platforms.Audit and reporting · Free
MedSafetyBenchMedical safety prompts for evaluating model safety in medical contexts.Datasets and benchmarks · Open source
PhishTankCommunity verified phishing URL database with a free API.Threat intelligence · Free
PKU-SafeRLHFPrompts marked for unsafe responses across multiple harm categories.Datasets and benchmarks · Open source