Member of Technical Staff - Pre-Training Evals Infra

Reflection is a research lab making intelligence open and accessible for everyone to use, customize, and build on. The role involves building the evaluation infrastructure that researchers use to understand model capabilities, guide training decisions, and assess readiness for release, spanning evaluation authoring, distributed execution, and results analysis, with a particular focus on pre-training. The role works directly with researchers to make evaluations faster, more reliable, and easier to use, ensuring the results accurately reflect the models being evaluated.

Responsibilities

Design and evolve evaluation abstractions, building clear APIs and data models for evaluation tasks, datasets, configurations, model outputs, and scores; Build scalable, reliable execution infrastructure, developing distributed systems for evaluating model checkpoints throughout training and ahead of release; Optimize evaluation and inference performance by profiling inference servers and distributed execution to improve throughput, GPU utilization, and time to results; Establish evaluation correctness and reproducibility through validation and regression tests, versioned datasets and configurations, and preserved provenance; Build researcher-facing tools, including interfaces and dashboards for inspecting model outputs, comparing checkpoints and experiments, and debugging unexpected behavior; Operate and monitor evaluations as training progresses, tracking workloads and results, triaging failures, and developing alerts and diagnostics; Turn research needs into reusable systems by partnering with researchers to identify bottlenecks and turn bespoke workflows into self-service tools

Requirements

Strong proficiency in Python, C++, or Rust, with the ability to reason deeply about concurrency, memory behavior, and performance; Experience designing, building, and operating reliable, maintainable distributed systems, with sound judgment about interfaces, failure modes, and architectural tradeoffs; Working knowledge of model inference and evaluation, including how implementation and configuration choices can affect measured model quality; Ability to investigate complex problems across system boundaries, follow evidence, and identify root causes rather than just address symptoms; Strong product judgment and experience translating the needs of researchers or other technical users into clear APIs and usable tools; Comfort owning ambiguous problems end to end, prioritizing competing research needs, and balancing rapid delivery with long-term reliability and maintainability; Attention to detail and the persistence to investigate subtle discrepancies in model outputs, evaluation results, and system behavior

Benefits

Top-tier compensation with salary and equity; Stock options; Comprehensive medical, dental, vision, and life insurance with an annual wellness allowance; Lunch and dinner provided in the office daily; 22 weeks paid parental leave for all new birthing and non-birthing parents, including adoptive and surrogate journeys; Unlimited paid time off in the U.S. and 30 days in the U.K.; Visa sponsorship support and long-term immigration pathways where applicable; Regular off-sites, happy hours, and team celebrations