Staff Applied AI Scientist

Shipping an AI product is only the beginning. The harder challenge, one few teams have mastered is continuous production evaluation: diagnosing performance shifts in real-time, decoding 'why' they occur, and driving measurable quality improvements at scale. We are looking for a Staff level Applied AI Scientist with a strong AI Engineering background to solve this problem for our Coach AI system, establishing the observability and evaluation frameworks that turn early production releases into robust, high-performance production products and then to make this sustainable by enabling the rest of our engineering org to do the same.

Responsibilities

Own the end-to-end feedback loop; Contribute to Context engineering; Design and run evals; Eval-driven agentic orchestration; Model and provider selection; Create and Monitor guardrails and safety in production; Enable others; Partner closely; Stay current

Requirements

Experience building and turning production agentic systems, including context engineering, RAG, memory, cost, model selection and performance; Proven experience analysing the performance of AI or data products in production and turning it into changes that maintained and improved the product; Hands-on LLM evaluation in production: LLM-as-judge, eval datasets, human-in-the-loop labelling, scoring against thresholds; Experience with Observability tooling for LLM and agentic systems (traces, sampling, prompt management, production monitoring such as Langfuse or comparable); Experience with longitudinal measurement: metrics and baselines, regression detection, quality tracking over time; AI-native daily practice, comfortable using agentic coding tools (Claude Code, Cursor, Codex or similar) on multi-step tasks, with clear judgment on when to direct an agent versus write code yourself; Strong technical writing and communication, and a track record of building capability into systems and teaching others to own it; Strong signals: built or scaled an eval and observability practice across multiple teams; evolved existing enterprise codebases with AI; production agentic systems (orchestration, RAG); a postgraduate degree in ML, CS, Applied Maths or related; public writing, talks or open-source work in eval, observability or LLMOps; Candidates must be legally authorised to work in Australia for the duration of employment

Benefits

Equity through Employee Share Option Program; Learning programs and coaching; Quarterly refresh days; extended end-of-year break and monthly allowance; Inclusive parental leave; MacBook and home workspace budget; Five annual social impact days; Medical insurance coverage for you and your family (Available for US & UK only)