Trust & Safety Tools and Services

AymaraAutomated evaluations for AI safety, accuracy and jailbreak vulnerability.Red teaming and evaluation · Open source
CounterfitAutomation for assessing the security and robustness of AI models.Red teaming and evaluation · Open source
DeepEvalOpen source LLM evaluation with pytest style tests and a red teaming module.Red teaming and evaluation · Open source
GarakFramework for adversarial testing and evaluation of language models.Red teaming and evaluation · Open source
GiskardOpen source testing for LLM and ML apps, with a scan for common vulnerabilities.Red teaming and evaluation · Open source
Granite GuardianOpen guardrail models for detecting harms in LLM traffic.AI guardrails · Open source
Guardrails AIPython framework for validating what goes into and comes out of a model.AI guardrails · Open source
InspectOpen evaluation framework for LLMs from the UK AI Security Institute, widely adopted.Red teaming and evaluation · Open source
Llama GuardOpen weight model that checks prompts and responses against a written policy.AI guardrails · Open source
LLM CanaryEvaluates models for security vulnerabilities and adversarial robustness.Red teaming and evaluation · Open source
NeMo GuardrailsToolkit for programmable rails around LLM apps: topic, safety, jailbreak and fact checks.AI guardrails · Open source
Prompt FuzzerTests prompt injection weaknesses in systems built on language models.Red teaming and evaluation · Open source
PromptfooAutomated model evaluations with ready made attack strategies and reports.Red teaming and evaluation · Open source
PyRITMicrosoft's Python toolkit for AI red teaming and security testing.Red teaming and evaluation · Open source
ShieldGemmaOpen weight safety classifiers for model inputs and outputs.AI guardrails · Open source
WildGuardOpen moderation model for prompts and responses, with a benchmark of real and adversarial cases.AI guardrails · Open source