Back to search
Responsibilities
- Measure abuse: build statistical telemetry and measurement frameworks to detect and monitor policy violations or emergent harms
- Scale model evaluation: design a scalable framework adapting to evolving agentic model capabilities and safety/risk landscape, grounded in statistics
- Evaluate and optimize safeguards: scale offline and online performance evaluation of our safeguards using human-in-the-loop and active learning
- Design experiments and analyses: Conduct controlled experiments and rollouts to evaluate the impact of policy or risk definition changes and safety mitigations
- Build intelligence and reporting: build signals, pipelines, dashboards to enable real-time monitoring driving interventions and safety roadmaps
Minimum Qualifications
- Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience
- Bachelor's degree in Mathematics, Statistics, Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience
- A minimum of 6 years of work experience in analytics (minimum of 4 years with a Ph.D.)
- Experience with data querying languages (e.g. SQL), scripting languages (e.g. Python), and/or statistical/mathematical software (e.g. R)
Preferred Qualifications
- Experience in frontier AI products or risks, and navigating online, adversarial environments in Trust & Safety or Fraud/Risk/Security domains. Model evals, threat modelling, actor telemetry, human-in-the-loop review systems don’t sound foreign to you
- Familiar with fast-paced, high-ambiguity, cross-functional environments – able to jump from agentic trace deep-dives to explaining risk dimensions in plain English to policy stakeholders
- Background in ambiguous and sparse data environments to operationalize e.g. harm prevalence measurement, causal inference, root-cause analysis– rooted in strong quantitative/statistical foundations
- Master's or Ph.D. Degree in a quantitative field
- Experience adhering to and implementing responsible, ethical AI practices (e.g., risk assessment, bias mitigation, quality and accuracy reviews)
- Demonstrated ability to integrate AI tools to optimize/redesign workflows and drive measurable impact (e.g., efficiency gains, quality improvements)
- Demonstrated ongoing AI skill development (e.g., prompt/context engineering, agent orchestration) and staying current with emerging AI technologies
$177,000/year to $247,000/year + bonus + equity + benefits
Learn more about this Employer on their Career Site
