TL;DR: You own how we measure and improve Lovable's AI agent. You build the eval systems and experiments that tell us whether a change makes the agent better or worse, and you turn agent telemetry into the fixes that raise success rates and cut errors. At Lovable, data scientists are not isolated model-builders; they sit close to the product, experimenting continuously with how intelligence changes user behavior and product dynamics.
Responsibilities
Define and own the metrics for agent quality: success, completion, error rates, and the behaviors that drive them; Build the eval systems and experiment framework that decide whether an agent change ships, such as an A/B-tested rollout; Turn agent traces and telemetry into concrete fixes, working directly with the agent engineering team; Build the tooling and agents that produce evaluations continuously as the agent evolves; Set the bar for how to judge agent behavior where there is no answer key to check againstRequirements
Experience or strong interest in LLM evaluation and observability: building evals, scoring outputs, tracing agent behavior, and catching regressions; Strong SQL and Python; Applied statistics and experimentation; Comfortable designing A/B tests for agent changes where outcomes are noisy; Ability to build systems and agents that produce insight continuously rather than one-off analyses; Instinct for what "good" looks like in agent behavior (success, error rates, task completion) and how to measure it when there is no clean answer key; Entrepreneurial, thrives in ambiguity, and works closely with engineersHow to apply
Apply online: https://www.jobs.pl/en/oferta/103450641/aplikuj
JOBS.pl · https://www.jobs.pl/data-scientist-agent-oferta-103450641





