Senior ML Systems Engineer, Inference

Runpod is the AI Developer Cloud. We're looking for a ML Systems Engineer, Inference. We want Runpod to be the best place in the world to run LLM inference, meaning the fastest and the most cost-efficient. You'll lead that effort and own LLM serving performance end to end: measuring it, understanding it, and improving it across models, hardware generations, and workloads. This is a hands-on engineering role for someone who likes finding the real bottleneck and fixing it, then turning that fix into something that runs reliably in production.

Responsibilities

Define how to measure inference performance, including throughput, time to first token, inter-token latency, and cost per token, and build the tooling that makes those measurements rigorous and repeatable; Profile and diagnose performance problems across the serving stack, from scheduling and memory management down to kernels and interconnect; Improve serving efficiency for large, state-of-the-art models on single-node and multi-node GPU deployments; Turn what is learned into production-ready runtimes, configurations, and defaults that customers benefit from automatically; Work closely with product and infrastructure teams to shape how inference is offered on Runpod; Keep up with the fast-moving inference ecosystem, including the open-source community, and decide what is worth adopting, building, and contributing back; Trace bottlenecks in the serving engine/runtime and implement fixes when configuration tuning is not enough

Requirements

5+ years of professional system engineering experience; Deep, hands-on experience with vLLM, SGLang (or a comparable serving engine) in production or at serious benchmark scale; Strong software engineering skills in Python, comfortable working in large, performance-critical codebases; A solid understanding of what drives LLM inference performance: batching, memory, parallelism, and the trade-offs between latency and throughput; Experience with modern inference optimization techniques such as quantization, speculative decoding, or distributed serving; Rigor in benchmarking and performance analysis, plus comfort with GPU profiling tools; The ability to explain results clearly in writing and turn them into decisions

Nice to have

Experience writing or tuning GPU kernels in CUDA or Triton; Contributions to inference or ML systems projects; Experience with multi-node GPU systems and high-speed networking; Experience at a company where inference cost and latency were core business metrics

Benefits

Meaningful equity in the form of stock options for everyone on the team; Generous medical, dental & vision plans; Flexible PTO; Most roles are remote-first with an inclusive, collaborative team using Slack as the main internal communication tool; $1,200 Home Office & Equipment Stipend