AI Model Policy Trainer, Generalist (Seattle)

Written by the T&S Network from Handshake's posting.

Policy evaluation work for AI models: trainers read user requests and model responses, then pick the policy category that fits each one. The work covers a customer's policy taxonomy, with subject matter that shifts by project, including harm facilitation, emotional reliance, and sexual-safety boundaries. The team calibrates classifications and rationales with evaluators, project leads, policy teams and researchers.

• Adult and sexual content
• Self harm and suicide

What you'd do

• Read user requests, model responses and prior conversation history before choosing a policy label.
• Pick between closely related labels, severity levels and policy boundaries, and document the choice.
• Write short, evidence-based rationales that cite policy language and specific conversation details.
• Point out where policy text is incomplete, conflicts with itself or misses a new case, and escalate unresolved cases as questions.
• Take part in calibration discussions and revise your judgment when new evidence comes up.
• Keep labeling accuracy and consistency steady across repetitive evaluation work.
• Help refine rubrics, examples, decision rules and quality standards.

What it asks for

• Strong analytical reasoning and the ability to apply detailed written criteria consistently.
• Ability to write clear rationales that another reviewer can audit.
• Background in QA, research, editing, law, trust and safety, content moderation, compliance or similar careful-judgment work (preferred).
• Experience evaluating language models, AI annotation, RLHF or content moderation (preferred).
• Experience applying rubrics, taxonomies or regulatory language (preferred).
• Comfort with long conversations, incomplete context and conflicting evidence (preferred).

Worth knowing

• Regular exposure to sexual content, emotional distress, self-harm, suicide, violence, weapons, abuse and exploitation material.
• Full-time, 40 hours per week, employed through TCWGlobal as a W-2 assignment supporting Handshake AI.

Terms in this role

RLHF.

Reinforcement learning from human feedback: training an AI model on people's ratings of its answers.

Our best summary of the posting, and it can be wrong. The employer's posting is what counts: read it in full on Handshakenorth_east before you apply. If we got something wrong, report this listing.