AI Evaluation Engineer

In the Engineering group, we are accelerating the product vision by building solutions in a fun, collaborative, efficient, and adaptive team. As an AI Evaluation Engineer, you will own the quality bar for Yes AI — defining what "correct" looks like when an AI answers a complex power market question, and building the frameworks to measure it at scale. This is a foundational role at the intersection of LLM engineering, domain expertise, and rigorous testing. You will also work directly on projects to integrate systems and serve energy data to customers. We are curious people who take pride in our work, value high-quality communication, and strive to improve continuously.

Responsibilities

Own the evaluation strategy for Yes AI end-to-end — design, build, and continuously improve frameworks that measure the quality, accuracy, and reliability of LLM-powered outputs across natural language answers, data retrieval, and semantic context responses; Develop benchmarks and automated test suites to detect hallucination, factual drift, and retrieval failures; set the quality bar for what "correct" means in power market AI; Partner directly with domain experts and senior engineers to translate deep power market knowledge into rigorous, repeatable evaluation criteria; Build and maintain scalable eval pipelines that run continuously against prompt changes, model updates, and Snowflake data changes — surfacing regressions before they reach customers; Lead prompt engineering and RAG experimentation efforts, using evaluation results to drive measurable improvements in answer quality; Influence engineering and product decisions by making AI quality visible and quantifiable — advocate for reliability as a first-class product requirement; Help define the discipline of AI evaluation at Yes Energy; this is a foundational role with room to build process and tooling from scratch

Requirements

Bachelor's degree in Computer Science, Data Science, Mathematics, or related field; 5+ years of software engineering experience, with significant focus on testing, evaluation, quality systems, or AI/ML tooling; Strong Python skills; experience building production-grade evaluation or testing pipelines; Solid understanding of LLM evaluation concepts: benchmarking, hallucination detection, retrieval-augmented generation quality, and semantic similarity; Experience designing and owning quality frameworks in ambiguous, fast-moving environments; Ability to work closely with non-engineer subject matter experts to define and codify what "good" looks like; Excellent judgment — knows when an eval result signals a real problem vs. noise; Strong communicator; able to present quality findings clearly to technical and non-technical audiences

Nice to have

Technical Execution & Engineering Tech: Hands-on experience with eval frameworks (RAGAS, LangSmith, PromptFlow) and proficiency in building production data pipelines using RAG or vector databases; Problem Solving: Applies structured thinking and sound judgment to determine when an eval result signals a real problem versus noise in complex data sets; Curious: Demonstrates an innate curiosity and a strong desire to learn the energy industry or apply experience from equities/commodities trading environments; Stakeholder Focus: Rooted in listening to technical and non-technical business partners to define and translate deep operational requirements into codified test criteria; Data Management & Quality: Background in data-intensive or highly regulated environments where answer accuracy is high-stakes and strict data governance is required; Effective Communication: Strong written and verbal communication skills, adjusting delivery style seamlessly based on the technical depth of the audience

Benefits

Competitive salaries; achievable bonuses; private medical insurance; wellness/gym benefits; flexible vacation; flexible work schedules; funded formal and informal professional development