Senior AI Evaluation Engineer (CICD & Platform)

We are seeking a Senior AI Evaluation Engineer (CI/CD & Platform) to own the continuous evaluation architecture, deployment pipeline infrastructure, and release gating for our enterprise AI Skills platform. In this role, you will build both the MLOps/CI-CD deployment pipelines and the automated eval harnesses that ensure all production AI skills execute accurately, remain schema-compliant, and run safely in live customer environments. You will bridge the gap between AI skill development and production platform engineering, leveraging modern evaluation frameworks (e.g., LangSmith, Braintrust, DeepEval) alongside CI/CD pipelines to evaluate agent trajectories, test schema contracts, and certify every skill before deployment.

Responsibilities

Continuous Evaluation: build tiered evaluation pipelines separating instant static/schema checks for PR developer loops from comprehensive synthetic, multi-turn trajectory, and LLM-as-a-Judge eval suites for release promotion gates; CI/CD Quality Gates & MLOps Infrastructure: architect and maintain CI/CD build pipelines (GitHub Actions, GitLab CI), skill registry versioning, and runner environments to enforce automated pass/fail release gates on every pull request; Deterministic Code & Schema Validation: develop synthetic data generation pipelines and automated red-teaming utilities that allow skill authors to generate edge-case evaluation suites from raw API schemas and domain specifications; Benchmark Datasets & Regression Testing: curate versioned golden datasets and execute continuous regression testing across base model upgrades, prompt changes, and security boundary checks; Production Observability: ingest real-world production traces and failure logs back into benchmark datasets to continuously close test coverage gaps

Requirements

4+ years of professional software engineering experience in Python building production-grade systems, with hands-on strength in either CI/CD pipeline engineering or AI evaluation methodology, plus working familiarity with the other; 2+ years designing, building, and operating test automation frameworks (pytest), CI/CD build pipelines (GitHub Actions, GitLab CI), or automated release gates; 1+ years of hands-on experience evaluating or building Generative AI applications (LLM tool-calling, agent workflows, or structured output parsing); Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience

Nice to have

AI Evaluation Tooling: direct experience with modern evaluation and observability frameworks (e.g., LangSmith, Braintrust, DeepEval, Promptfoo, Phoenix); Schema Contracts & API Validation: strong knowledge of schema validation libraries (JSON Schema, Pydantic, OpenAPI) and API contract testing; Agentic Standards & MCP: familiarity with the Model Context Protocol (MCP), tool-calling specifications, or agent execution harnesses; Static Code Analysis: experience using schema validation frameworks, linters, and test automation tooling to enforce code contracts; AI Security & Governance: understanding of core LLM failure modes (hallucinations, prompt injection, PII/PHI sanitization, RBAC boundaries); Quality & Testing Discipline: strong engineering rigor around test coverage, dataset versioning, evaluator calibration metrics, and regression prevention in continuous deployment pipelines; Technical Communication & Leadership: ability to own problem domains end-to-end, collaborate with platform and product teams, and write clear technical documentation and architecture designs

Benefits

Workday Bonus Plan or role-specific commission/bonus; annual refresh stock grants awarded by Workday Inc.; comprehensive benefits; Flex Work arrangement combining in-person and remote time