ML Systems Engineer - Model Training and Infrastructure (SWE-focused LLMs)

We're looking for an ML Systems Engineer to collaborate in training our Lumen models, our open-source-based software engineering LLMs. This is an interdisciplinary role involving developing and deploying RL training environments, working on large-scale synthetic data pipelines, and building the ML platform to enable training large models at scale in distributed environments. The goal is to train models to be the best SWEs in the world, writing readable, maintainable code that fits the architectural patterns already present in the codebase. The role is full in-office, five days a week, with a start date ASAP.

Responsibilities

Participate in end-to-end training of models, including supervised fine-tuning on curated code and conversation datasets and RL on top of those models to align them with software-engineering objectives; Develop and manage synthetic data generation pipelines to curate datasets for future training runs; Design, build and deploy containerized services using Docker and platforms like Kubernetes to enable RL infrastructure; Build and maintain the training platform for model architecture experiments at scale, model training, and evaluations; Ideate on novel and opinionated reward functions for training SWE agents; Collaborate with infra, product, and research to decide what to train next, how to train it, and how to measure improvement for engineers; Develop and deploy RL training environments and build the ML platform for training large models in distributed environments

Requirements

Typically 3-5 years of experience in software engineering or computer science; Ability to read, debug, and write non-trivial production code (mainly Python and Go); Experience with Docker and container management/orchestration platforms such as Kubernetes; Experience with at least one major cloud computing platform such as GCP, AWS or Azure; Care about code quality, correctness, and maintainability as much as model metrics; Comfortable working with large-scale datasets, object storage, dataset sharding, and filtering; Understanding that data quality and sampling strategies matter as much as architecture; Able to turn a vague modelling goal into a concrete plan of experiments; Comfortable documenting decisions and walking others through tradeoffs

Nice to have

Knowledge of PyTorch/Tensorflow/JAX; Experience with synthetic data generation pipelines; Experience with data tooling such as SQL, Apache Iceberg and duckDB; Experience training LLMs in distributed environments; Experience with LLM-as-a-judge, reward hacking detection, or robustness evaluation; Open-source contributions or research, such as contributions to open-source LLM tooling or RL libraries

Benefits

Competitive salary, benchmarked to the market; Equity / share options; 30 days' holiday + bank holidays; Dog-friendly office; Weekly team breakfast & lunch; Monthly socials; Pension; High-quality equipment