Tools & Services

AI Misuse and Jailbreaks

People turning an AI model to harm: jailbreaks, prompt injection, and getting a model to produce what its policy forbids.

41 entries

Most upvotedA to Z
  • AILuminateHuman written prompts across harm categories, from MLCommons.Datasets and benchmarks · Open source
  • ALERTStandard and adversarial red teaming prompts, from Babelscape.Datasets and benchmarks · Open source
  • Alice AI Red TeamingExpert-led and automated red teaming of foundation models across text, image, audio and video.Red teaming and evaluation · Service
  • Arthur ShieldLLM firewall and evaluation from an ML observability vendor.AI guardrails · Enterprise
  • Aya Red-teamingMultilingual red teaming prompts across harm categories, from Cohere Labs.Datasets and benchmarks · Open source
  • AymaraAutomated evaluations for AI safety, accuracy and jailbreak vulnerability.Red teaming and evaluation · Open source
  • Bugcrowd AI Safety and SecurityCrowdsourced AI red teaming, AI penetration tests, bias assessments and bug bounties for AI systems.Red teaming and evaluation · Service
  • CounterfitAutomation for assessing the security and robustness of AI models.Red teaming and evaluation · Open source
  • DeepEvalOpen source LLM evaluation with pytest style tests and a red teaming module.Red teaming and evaluation · Open source
  • DEF CON Red Teaming DatasetResults from DEF CON's generative AI red teaming event.Datasets and benchmarks · Open source
  • Do Not AnswerQuestions across risk areas for testing safety and refusal behavior.Datasets and benchmarks · Open source
  • GarakFramework for adversarial testing and evaluation of language models.Red teaming and evaluation · Open source
  • Gemini API safety settingsConfigurable harm filters built into the Gemini API.AI guardrails · Free
  • GiskardOpen source testing for LLM and ML apps, with a scan for common vulnerabilities.Red teaming and evaluation · Open source
  • Granite GuardianOpen guardrail models for detecting harms in LLM traffic.AI guardrails · Open source
  • Guardrails AIPython framework for validating what goes into and comes out of a model.AI guardrails · Open source
  • HackAPrompt DatasetPrompt based jailbreak attempts, for testing model vulnerability.Datasets and benchmarks · Open source
  • HackerOne AI Red TeamingTime-boxed adversarial testing of AI models, agents and apps by vetted outside researchers.Red teaming and evaluation · Service
  • HarmBenchEvaluation dataset for automated red teaming, from the Center for AI Safety.Datasets and benchmarks · Open source
  • Humane Intelligence Red TeamingDesigns and runs public and expert red teaming exercises for AI systems as a paid service.Red teaming and evaluation · Service
  • Innodata Model Safety and Red TeamingHuman and automated red teaming of generative AI models, plus 24/7 content moderation.Red teaming and evaluation · Service
  • InspectOpen evaluation framework for LLMs from the UK AI Security Institute, widely adopted.Red teaming and evaluation · Open source
  • JailbreakBenchHarmful behaviors used as a benchmark for jailbreaking evaluation.Datasets and benchmarks · Open source
  • Lakera GuardHosted detection of prompt injection, jailbreaks and data leakage for LLM apps.AI guardrails · Usage based
  • LangSmith EvaluationEvaluation and tracing for LLM apps built with or without LangChain.Red teaming and evaluation · Freemium
  • Llama GuardOpen weight model that checks prompts and responses against a written policy.AI guardrails · Open source
  • LLM CanaryEvaluates models for security vulnerabilities and adversarial robustness.Red teaming and evaluation · Open source
  • LXT LLM Red Teaming and Safety EvaluationHuman-led adversarial testing of LLMs for jailbreaks, toxicity, bias and hallucination.Red teaming and evaluation · Service
  • NeMo GuardrailsToolkit for programmable rails around LLM apps: topic, safety, jailbreak and fact checks.AI guardrails · Open source
  • OWASP Top 10 for LLM ApplicationsThe reference list of security risks in LLM applications.Red teaming and evaluation · Free
  • PKU-SafeRLHFPrompts marked for unsafe responses across multiple harm categories.Datasets and benchmarks · Open source
  • Prompt FuzzerTests prompt injection weaknesses in systems built on language models.Red teaming and evaluation · Open source
  • Prompt SecurityRuntime protection for GenAI apps and employee GenAI use.AI guardrails · Enterprise
  • PromptfooAutomated model evaluations with ready made attack strategies and reports.Red teaming and evaluation · Open source
  • Protect AIAI security platform: LLM firewall, model scanning and red teaming, now part of Palo Alto Networks.AI guardrails · Enterprise
  • PyRITMicrosoft's Python toolkit for AI red teaming and security testing.Red teaming and evaluation · Open source
  • ShieldGemmaOpen weight safety classifiers for model inputs and outputs.AI guardrails · Open source
  • SorryBenchAdversarial prompts with linguistic mutations for testing safety.Datasets and benchmarks · Open source
  • SOSBenchRegulation grounded prompts across six high risk scientific domains.Datasets and benchmarks · Open source
  • Trail of Bits AI/ML SecuritySecurity assessments of AI systems end to end, including model capability evaluations.Red teaming and evaluation · Service
  • WildGuardOpen moderation model for prompts and responses, with a benchmark of real and adversarial cases.AI guardrails · Open source

Keep a list of the ones worth a call

A free account lets you upvote these and keep them in a list of your own. We never pass your details to a vendor.

Create a free account