Datasets and Benchmarks
Labeled data to train a classifier on or test one against: toxicity corpora, adversarial prompt sets, and harm taxonomies somebody has already annotated.
20 tools
Aegis Content SafetyNVIDIA's dataset for content moderation and toxicity detection.Datasets and benchmarks · Open source
Jigsaw ToxicityWikipedia comments labeled by human raters for toxic behavior.Datasets and benchmarks · Open source
AILuminateHuman written prompts across harm categories, from MLCommons.Datasets and benchmarks · Open sourceALERTStandard and adversarial red teaming prompts, from Babelscape.Datasets and benchmarks · Open source
Aya Red-teamingMultilingual red teaming prompts across harm categories, from Cohere Labs.Datasets and benchmarks · Open source- badwordsWord lists in several locales for flagging user submitted content.Datasets and benchmarks · Open source
DarkBenchBenchmark for detecting dark design patterns in language models.Datasets and benchmarks · Open source
DEF CON Red Teaming DatasetResults from DEF CON's generative AI red teaming event.Datasets and benchmarks · Open source
Do Not AnswerQuestions across risk areas for testing safety and refusal behavior.Datasets and benchmarks · Open sourceHackAPrompt DatasetPrompt based jailbreak attempts, for testing model vulnerability.Datasets and benchmarks · Open source
HarmBenchEvaluation dataset for automated red teaming, from the Center for AI Safety.Datasets and benchmarks · Open source
JailbreakBenchHarmful behaviors used as a benchmark for jailbreaking evaluation.Datasets and benchmarks · Open source- MedSafetyBenchMedical safety prompts for evaluating model safety in medical contexts.Datasets and benchmarks · Open source
PKU-SafeRLHFPrompts marked for unsafe responses across multiple harm categories.Datasets and benchmarks · Open sourceSorryBenchAdversarial prompts with linguistic mutations for testing safety.Datasets and benchmarks · Open source
- SOSBenchRegulation grounded prompts across six high risk scientific domains.Datasets and benchmarks · Open source
Toxic ChatToxic conversations collected from real interactions with a chat assistant.Datasets and benchmarks · Open source
ToxiGenLarge machine generated dataset of implicit hate speech across 13 minority groups.Datasets and benchmarks · Open source
Uli DatasetGendered abuse data, created for Tattle's Uli redaction model.Datasets and benchmarks · Open source- XSTestPrompts that test exaggerated safety behavior and over refusal.Datasets and benchmarks · Open source
What to ask before you buy
- What is your recall and precision on my harms, in my languages, at my volume? Not in general.
- What happens when you're wrong, and how does a person override it?
- Where does my data sit, who can read it, and what happens to it when we leave?
- What does the price do when our volume doubles? Show me the tier boundaries.
- Who else my size uses this, and can I speak to one of them without you present?
- What is your incident process when your service is down and my queue is filling?
Rules that push teams here
Not legal advice
- The Digital Services ActRisk assessment, transparency reporting, and a working appeals route.
- The Online Safety ActDuties that scale with your size and your users' age.
Keep a list of the ones worth a call
A free account lets you upvote these and keep them in a list of your own. We never pass your details to a vendor.
Create a free account