Safeguards Enforcement Analyst, Cyber Harm

Written by the T&S Network from Anthropic's posting.

Enforcement review of flagged activity on Anthropic's AI products, aimed at detecting and mitigating attempts to misuse the models for malicious cyber operations. The initial focus is cyberattacks, malware development, and offensive exploitation, with the role possibly expanding to broader enforcement areas. The work sits within the Safeguards team and feeds policy design and detection model improvement.

What you'd do

• Review flagged content and accounts and make documented enforcement decisions under the usage policies.
• Detect and mitigate misuse of AI systems for cyberattacks, malware creation, and exploitation tooling.
• Give the Safeguards policy design team feedback on gaps surfaced by real enforcement cases.
• Send detection model errors and quality signals to Engineering and Data Science to improve precision and recall.
• Keep accuracy and consistency high across review queues.
• Track threat actor tactics and the evolving cyber threat landscape to inform enforcement calls.

What it asks for

• Pass unclear, new, or severe cases up to the people who should decide them.
• Background in offensive security, exploit writing, malware reverse engineering, or bug research.
• Track record of moderating or investigating abuse at high volume.
• Working knowledge of SQL or Python for analyzing data and spotting threats.
• Skill at flagging emerging risks and explaining them to non-specialist teams like Product and Legal.
• Hands-on use of generative AI tools, including crafting prompts for review work.
• Experience in trust and safety, abuse or cyber investigations, or threat intelligence at a technology or AI company (preferred).

Worth knowing

• Exposure to explicit content, including violent, technical, or psychologically disturbing material.
• Responding to escalations during weekends and holidays.

Our best summary of the posting, and it can be wrong. The employer's posting is what counts: read it in full on Anthropicnorth_east before you apply. If we got something wrong, report this listing.