Research, Evals

Exa is an applied AI lab building a search engine unlike the world has ever seen. We're looking for an ML evals engineer to design and build our eval stack at Exa. The role involves investigating how to evaluate search engines in an LLM world and then building the most comprehensive, creative, and effective eval suite.

Responsibilities

Design and build our eval stack at Exa; Investigating how to evaluate search engines in an LLM world; Building the most comprehensive, creative, and effective eval suite; Write a manifesto of what perfect search means; Design and implement evaluation frameworks that probe the limits of search; Build scalable, reliable eval pipelines that track regressions, drift, and quality signals across billions of documents; Create golden datasets, synthetic benchmarks, agentic tasks, and real-world test suites that reflect how developers, agents, and humans actually use Exa; Partner closely with ML researchers, data engineers, infra engineers, and product to shape the feedback loops that improve our search models

Requirements

Have hands-on ML experience (training, finetuning, or evaluating models); Have strong engineering fundamentals and can build reliable systems (Python, Rust, distributed pipelines, GPU/cluster jobs, etc.); Enjoy diving into data via building eval sets, inspecting edge cases, designing creative measurement strategies

Nice to have

embeddings; LLMs

Benefits

Premium healthcare benefits (medical, dental, vision); Fertility benefits; 16 weeks of fully paid parental leave for all new parents; Monthly wellness stipend