Trust & Safety Tools and Services

AILuminateHuman written prompts across harm categories, from MLCommons.Datasets and benchmarks · Open source
ALERTStandard and adversarial red teaming prompts, from Babelscape.Datasets and benchmarks · Open source
Aya Red-teamingMultilingual red teaming prompts across harm categories, from Cohere Labs.Datasets and benchmarks · Open source
AymaraAutomated evaluations for AI safety, accuracy and jailbreak vulnerability.Red teaming and evaluation · Open source
CounterfitAutomation for assessing the security and robustness of AI models.Red teaming and evaluation · Open source
DeepEvalOpen source LLM evaluation with pytest style tests and a red teaming module.Red teaming and evaluation · Open source
DEF CON Red Teaming DatasetResults from DEF CON's generative AI red teaming event.Datasets and benchmarks · Open source
Do Not AnswerQuestions across risk areas for testing safety and refusal behavior.Datasets and benchmarks · Open source
GarakFramework for adversarial testing and evaluation of language models.Red teaming and evaluation · Open source
GiskardOpen source testing for LLM and ML apps, with a scan for common vulnerabilities.Red teaming and evaluation · Open source
Granite GuardianOpen guardrail models for detecting harms in LLM traffic.AI guardrails · Open source
Guardrails AIPython framework for validating what goes into and comes out of a model.AI guardrails · Open source
HackAPrompt DatasetPrompt based jailbreak attempts, for testing model vulnerability.Datasets and benchmarks · Open source
HarmBenchEvaluation dataset for automated red teaming, from the Center for AI Safety.Datasets and benchmarks · Open source
InspectOpen evaluation framework for LLMs from the UK AI Security Institute, widely adopted.Red teaming and evaluation · Open source
JailbreakBenchHarmful behaviors used as a benchmark for jailbreaking evaluation.Datasets and benchmarks · Open source
Llama GuardOpen weight model that checks prompts and responses against a written policy.AI guardrails · Open source
LLM CanaryEvaluates models for security vulnerabilities and adversarial robustness.Red teaming and evaluation · Open source
NeMo GuardrailsToolkit for programmable rails around LLM apps: topic, safety, jailbreak and fact checks.AI guardrails · Open source
PKU-SafeRLHFPrompts marked for unsafe responses across multiple harm categories.Datasets and benchmarks · Open source