Tech Lead Manager, AI Evaluations

Glean is the Work AI platform that helps everyone work smarter with AI. The role leads a team of applied AI scientists and engineers to build the evaluation strategy and platform that helps Glean ship reliable AI experiences across models, harnesses, assistants, agents, and other AI products. The role is hybrid (4 days a week in the Mountain View or San Francisco office).

Responsibilities

Own the AI evaluation engineering roadmap and technical direction end to end; build scalable, reusable evaluation infrastructure for datasets, experiments, graders, trace capture and replay, result analysis, and dashboards; create and maintain representative evaluation sets, including edge cases, adversarial inputs, and held-out acceptance sets; define a balanced quality measurement system covering correctness, groundedness, retrieval relevance, instruction following, citation quality, and task success, including agent trajectory evaluation; develop a mix of deterministic checks, model-based graders, and human evaluation, and calibrate automated judges; integrate evaluation into CI, pre-release suites, and sampled production evaluation; make quality tradeoffs measurable against latency, reliability, and cost; build and lead a high-performing team of applied AI scientists and engineers; partner with product, infrastructure, security, and application teams

Requirements

10+ years of experience in software engineering, ML infrastructure, data systems, or a related technical field; experience building and operating production systems; 5+ years of engineering or applied AI team management experience; hands-on experience designing, building, or operating evaluation systems for LLM/GenAI products, including model-based evaluators or autograders; familiarity with evaluation methods and tradeoffs (curated and production-derived datasets, rubric-based and pairwise judgments, human review, judge calibration, statistical analysis); understanding of agent evaluation (tool calling, multi-turn behavior, trajectory quality, task completion, recovery from errors); strong coding ability in Python or another production language; sound engineering fundamentals in testing, APIs, data modeling, and reliable distributed systems; strong judgment about measurement quality; product-minded and collaborative, able to communicate findings to technical and non-technical audiences; MS/PhD in Computer Science, AI, or a related field

Nice to have

Experience with safety testing or red teaming; knowledge of public benchmarks across domains and capabilities; familiarity with cloud infrastructure, Kubernetes, data stores, and observability

Benefits

Competitive compensation; variable compensation, equity, and benefits may be available for certain roles; Medical, Vision, and Dental coverage; generous time-off policy; 401k plan contribution opportunity; home office improvement stipend; annual education stipend; annual wellness stipend; regular company events; healthy lunches daily