AI Benchmark & Datasets Engineer / Researcher

About Pathway: Pathway builds the first post-transformer frontier model that solves AI's fundamental memory problem. While transformers wake up in the same state every time—like Groundhog Day—our architecture enables true continuous learning, infinite context reasoning, and real-time adaptation. We're not optimizing yesterday's technology; we're building what comes after transformers.

The Opportunity: You will design and execute rigorous benchmarks and define dataset standards. Collaborating closely with our R&D team, you will build the evaluation infrastructure that guides the evolution of Pathway’s post-transformer models.

Responsibilities

Proactively identify, prioritize, and curate relevant public and client-driven benchmarks across our target use cases and markets; Evaluate candidate benchmarks for clarity, data quality, evaluation methodology, and fit with our model roadmap; Run benchmarks with baseline models to validate setup, uncover edge cases, and de-risk R&D runs; Hand off “benchmark-ready” packages to R&D (specs, data, evaluation scripts, expected metrics, constraints); Maintain a shared vocabulary and documentation around benchmarks, datasets, and evaluation formats that GTM and R&D can both use; Track and organize benchmark results, model leaderboards, and “what good looks like” for different customers and scenarios; Contribute to demos and public-facing proof points based on benchmark outcomes

Requirements

Cover letter (2-3 lines); At least one of: published at least one paper at NeurIPS, ICLR, or ICML where you were the lead author or made significant conceptual & code contributions; significantly contributed to an LLM training effort which became newsworthy (topped a Hugging Face benchmark, best in class model, etc.), preferably using multiple GPUs; spent at least 6 months working in a leading Machine Learning research center (e.g. Google Brain/DeepMind, Apple, Meta, Anthropic, Nvidia, MILA); were an ICPC World Finalist, or an IOI, IMO, or IPhO medalist in High School; Experience with ML/LLM evaluation, data science, or technical product roles, ideally around benchmarks or experimentation; Comfortable reading papers, leaderboards, and Github repos, and turning them into clear, repeatable benchmark specs; Can talk comfortably with both engineers and customers, and translate between technical detail and business value; Care about high-quality data, reproducible experiments, and crisp documentation; Respectful of others; Fluent in English

Nice to have

Published or open-sourced work on LLM evaluation, benchmarking or data quality; Experience designing custom benchmarks or evaluation protocols for novel model capabilities

Benefits

Join an intellectually stimulating work environment; Be a pioneer: you get to work with a new type of 'Live AI' challenges around long sequences and changing data; Be part of an early-stage AI startup that believes in impactful research and foundational changes; Remote work; Possibility to work or meet with other team members in one of our offices: Palo Alto, CA; Paris, France or Wroclaw, Poland