Product Manager, Evals

Exa is an applied AI lab building a search engine. The Evals Team trains foundational models for search and measures what good search means. As a Product Manager on the Evals team, you will partner with research, product, GTM, and engineering to oversee the quality and completeness of evaluation data, manage external vendor relationships, and decide which data to acquire for evals. This is an in-person opportunity in San Francisco. Visa sponsorship is available for international candidates.

Responsibilities

Oversee the quality and completeness of evaluation data; Ensure search models are evaluated against comprehensive and high-quality datasets across numerous domains; Manage relationships with external vendors; Make key strategic decisions on what data is important for evals and acquire it; Build specific, localized eval feedback and signal; Write a manifesto of what perfect search means; Define metrics and success criteria for evaluation data suites; Partner with ML researchers, data engineers, infra engineers, and product to shape data loops that improve search models

Requirements

Hustler in negotiating, securing, and managing relationships with external vendors; Care and excellence in writing; High ownership and attention to detail; Strong numerical literacy and technical background; CS/EE degree, production engineering experience, or equivalent depth earned on the job

Benefits

Premium healthcare benefits (medical, dental, vision); Fertility benefits; 16 weeks of fully paid parental leave for all new parents; Monthly wellness stipend