Research Engineer, Evals

Written by the T&S Network from Variance's posting.

Builds the benchmarks, datasets, tooling and evaluation loops that measure how well AI agents make judgment calls in fraud, identity and risk investigations. The team is building agents for high-stakes gray-area decisions in messy, adversarial environments, and this role defines how model quality is measured and improved.

• Fraud and scams

What you'd do

• Build proprietary benchmarks and datasets for fraud, identity and risk workflows
• Design and run offline and online evals on real customer tasks
• Set the measures for judgment quality, such as accuracy, calibration, consistency and abstention
• Analyze where models and agents fail and turn those failures into evals and training loops
• Build reusable evaluation tools that work across product surfaces
• Review outputs, grade edge cases and curate datasets by hand until evals measure the right thing

What it asks for

• Experience training, evaluating or improving modern ML systems (preferred)
• Strong programming skills and comfort in research-heavy codebases (preferred)
• Have built benchmarks, datasets, eval pipelines or quality systems before (preferred)
• Familiarity with LLMs, agent systems, retrieval or post-training (preferred)
• Care about fraud, risk, trust and safety, compliance or other regulated and adversarial fields (preferred)

Our best summary of the posting, and it can be wrong. The employer's posting is what counts: read it in full on Variancenorth_east before you apply. If we got something wrong, report this listing.