Principal Research Scientist

G2 is building independent, rigorous evaluations of AI agents on realistic business work, to help buyers choose with confidence and to set a trusted standard for the market. This is the most senior applied research role on that work, and it is hands-on. The role owns the research agenda for how agents get tested, ensures results mean what G2 says they mean, and turns them into product features and insights G2 publishes. The role works side by side with engineers and product managers, and the work goes into what G2 ships today. The role also mentors engineers, data scientists and researchers working on evaluation, reviews their methodology, and makes the standards repeatable by others.

Responsibilities

Own the evaluation methodology for how G2 evaluates AI agents; Contribute to today's product by working closely with engineers and product managers to build and improve evaluations; Design realistic evaluations that reflect the work businesses give AI agents; Develop sound measurement, including scoring approaches with automated checks, LLM-based judges and human review; Validate the evaluations themselves by auditing tasks and graders, quantifying noise, and establishing when differences between agents are real; Represent the results to buyers, customers, partners, and the research community; Mentor and multiply by managing and guiding researchers and engineers on evaluation design and measurement; Set and pursue a research agenda on evaluation problems; Stay current with new benchmarks, papers and evaluation practices; Share and collaborate by writing methodology documentation and, where appropriate, papers and public posts

Requirements

Experience working with evaluation design or dataset curation in multi-agentic systems; 5+ years building or evaluating ML systems, including hands-on work with LLMs or LLM-based agents; 2+ years experience managing small to medium teams preferred; A graduate degree (MS or PhD) in machine learning, NLP, computer science, statistics or a related field, or equivalent applied research experience; A track record of original ideas in machine learning, or clear improvements on existing ones, shown through first-author publications or projects you led end-to-end; The ability to own a research agenda: pick the problems worth solving, and carry long-running projects without being directed; A track record of shipping ML work in a product setting, working closely with engineers and product managers; Experience setting technical direction for others: mentoring researchers or engineers, reviewing methodology, and raising the quality of work you did not do yourself; Familiarity with how AI systems are benchmarked, and a clear view of how benchmark scores can mislead; Sound statistical judgment: experimental design, variance, significance and agreement between human and automated raters; Good judgment about trade-offs, including when a practical solution now serves customers better than a more complete one later; Clear writing, and the ability to explain methodology to non-specialists and to defend it to skeptical audiences

Nice to have

Research experience in agent evaluation, user simulation, or human and AI-based judging; Experience with conversational, voice, retrieval-based or tool-using AI systems; Contributions to open-source evaluation tools or public leaderboards; Familiarity with business software workflows; A considered view on the effects of AI technology on the people and markets it reaches; Publications at ACL, EMNLP, NeurIPS, ICML, ICLR, COLM or similar venues