JailbreakBench

JailbreakBench is a centralized benchmark for evaluating jailbreak attacks against large language models. Jailbreak attacks cause large language models to generate harmful, unethical, or otherwise unwanted content. Its leaderboards track the performance of attacks and defenses for various LLMs. Separate leaderboards cover open-source models and closed-source models.

It includes a repository of jailbreak artifacts, an evolving dataset of adversarial prompts that submissions to the benchmark are required to provide to ensure reproducibility. It also offers a standardized evaluation library with a defined threat model, system prompts, chat templates, and scoring functions. Its dataset, JBB-Behaviors, is composed of 100 distinct misuse behaviors divided into ten broad categories. The dataset is complemented with 100 benign behaviors for evaluating overrefusal rates for new models and defenses.

The benchmark was presented in a paper at the NeurIPS Datasets and Benchmarks Track in 2024. The paper's authors include Patrick Chao, Edoardo Debenedetti, Alexander Robey, and Eric Wong, among others.