Research Engineer - Benchmarks

Every week, someone asks which data provider is actually best: for company records, for finding the right person, for fresh job listings. Right now the answers come from the vendors themselves. You'll build the independent version. You'll run rigorous, automated, public benchmarks of the providers in Alexandria and beyond, and publish them weekly. The same results will feed straight back into Alexandria so it learns which provider to call for which job. This isn't our internal evals role. You're measuring the market, in public, where every number will be challenged by the vendors it ranks. You'll own it end to end: the datasets, the ground truth, the scoring, the harness, the weekly release, and the loop into Alexandria's provider selection. No one hands you a methodology. You write it, defend it, and ship it every week.

Responsibilities

Design and run head-to-head benchmarks of data providers against verified ground truth; build and maintain test datasets and ground truth, keeping them from going stale or leaking; own the automated harness and the weekly benchmark release, ensuring every run is reproducible, versioned, and defensible; measure accuracy, coverage, freshness, speed, and cost; work with marketing to ship public leaderboard pages; close the loop into Alexandria so benchmark results change which provider gets called for what; keep expanding the categories benchmarked

Requirements

4+ years in ML, research engineering, or data engineering; evaluation or benchmark work you've shipped; shipped evals or benchmarks and can explain exactly why your results were trustworthy; experience at a frontier lab, data company, third-party benchmark org, or public benchmark project (or OSS contributions); strong in Python and API integrations; comfortable with messy vendor APIs, rate limits, and inconsistent schemas; knowledge of building test sets and scoring methods (sampling, labeling, inter-rater agreement, when to trust an LLM judge); uses AI heavily and keeps upgrading own workflow; writes findings clearly for engineers, marketers, and vendors; open to candidates already authorized to work in the US or Canada

Nice to have

Experience at a frontier lab, a data company (e.g., Scale, Surge, micro1, Mercor), a third-party benchmark org, or a public benchmark project; OSS contributors very welcome

Benefits

Competitive equity; generous PTO (15 days mandatory, more available on request, holidays excluded); 12 weeks fully paid parental leave; $100 USD/month wellness stipend; up to $1,000 USD/year learning and development expense; team offsites; 3 paid months sabbatical after 4 years; US: medical, dental, and vision coverage (100% employee, 50% partner and kids), life and disability insurance, virtual care, mental health benefits, fertility and family building, EAP, 401(k), pre-tax benefits (HSA, FSA, commuter), supplemental insurance options; Canada: extended health, dental, and vision (100% employer-paid for employee, partner, and kids), life and disability insurance, virtual care, mental health benefits, fertility and family building, group RRSP; SF perks: snacks, drinks, team lunches, e-bike loaner; Toronto perks: snacks, drinks, team lunches, transit card for GO Transit, subway, and streetcar