Senior Product Manager, Eval Quality, AI Data

Overview

You will own the product goal for evaluating AI Agents, shaping the framework and roadmap for high-quality, scalable evaluations. You’ll work at the intersection of product, research, and engineering to define methodologies that cover coverage, climbability, discriminability, and stability while preventing contamination. Collaborating across Gemini, AI2, and partner areas, you’ll translate research insights into practical product plans and day-to-day execution. This role combines strategic vision with hands-on delivery to advance safe and trustworthy agentic systems. A strong hook is the opportunity to define how the next generation of AI will be validated at scale.

Responsibilities

• Define the product goal and roadmap for AI agent evaluations (Everest, EvalHub)
• Synthesize research insights into actionable product plans and experiments
• Coordinate across Product Areas (Gemini, AI2, partners) to align priorities
• Drive day-to-day execution and cross-functional delivery in a fast-moving environment
• Engage with Executive Team Leads, Researchers, and senior leadership across time zones

Key requirements

• Bachelor’s degree or equivalent practical experience
• 8 years of experience in product management, engineering, and technology strategy
• 5 years of experience taking technical products from conception to launch

• exceptional written and verbal communication skills
• gravitas to align leadership across time zones
• ability to collaborate across Product Areas and teams
• Experience with LLM/GenAI evaluations
• post-training/mid-training RL/SFT workflows
• autorater or HiTL data collection

…