Senior AIOps Engineer

The future is bright for the Porch Group, and we'd love for you to be a part of it as our Senior AIOps Engineer I. In this role, you will partner with product managers, platform engineers, data scientists, and machine learning engineers to make sure our AI and ML-powered systems are reliable, observable, secure, and cost-efficient in production. You will focus on how AI systems run in the real world: monitoring model performance and drift, keeping deployment pipelines robust, managing incidents, and standing up new AI infrastructure. Our AI/ML stack is built on Python and runs on Kubernetes (GKE) and Google Cloud Platform, using tools such as Union Cloud (Flyte), BentoML, Feast, Label Studio, BigQuery, and Dataflow.

Responsibilities

Own production reliability for AI/ML services by monitoring availability and performance, defining SLOs and SLIs, and owning recurring model refresh cycles; Design and implement monitoring, logging, and alerting for models and data pipelines; Streamline deployment workflows (blue/green, canary, A/B, and shadow deployments) and improve CI/CD pipelines for AI services; Provision, deploy, configure, and maintain new AI infrastructure, including AI gateways, RAG platforms, LLM observability tools, agentic workflows, and no-code agent builders; Build automation and tooling to reduce manual operational work; Own the operational health of Label Studio, including supporting GCS buckets, BigQuery pipelines, and database and SSO dependencies; Stand up and operate RAG and vector database infrastructure; Monitor and optimize LLM token usage and costs across providers; Participate in an on-call rotation, triage and resolve production incidents, and lead post-incident reviews; Develop production-grade services and tooling, primarily in Python, on Kubernetes, and document standards, architecture diagrams, and operational playbooks

Requirements

4+ years of applicable experience, including at least 3 years in commercial software engineering or production operations (SRE, DevOps, Platform, or ML Platform); Experience operating services on Kubernetes and/or major cloud platforms; Experience operating production systems at scale, preferably distributed, microservices-based, or Kubernetes-based architectures; Hands-on experience with CI/CD tools and deployment automation (GitLab CI/CD, GitHub Actions, or similar); Experience with infrastructure-as-code tools (preferably Terraform); Experience with monitoring, logging, and alerting tools (Datadog, Prometheus, Grafana, ELK/EFK, or similar); Experience with container management and the Docker image lifecycle; Experience operating and administering self-hosted open-source tools on Kubernetes, including installation, configuration, upgrades, security hardening, and backup and disaster recovery; Understanding of basic machine learning concepts and model lifecycles (training, evaluation, deployment, monitoring), and familiarity with common ML tooling; Strong computer science and programming fundamentals, with solid problem-solving skills and ability to write high-quality software using Test-Driven Development (TDD), SOLID principles, clean code, and robust documentation; Ability to communicate effectively with data scientists, ML engineers, and product stakeholders; Comfort participating in an on-call rotation and working in a trustful, respectful, and collaborative environment; A curious and driven mindset, with a strong interest in how AI systems behave in production

Nice to have

Strong GCP experience preferred, including GKE, BigQuery, Pub/Sub, Vertex AI, Cloud SQL, GCS, and IAM; Familiarity with LLM ecosystem tooling (AI gateways, vector databases, RAG pipelines, embedding models, or LLM evaluation frameworks); Experience with Python-based data and ML stacks (NumPy, SciPy, Pandas, Scikit-Learn, PyTorch); Experience building or operating ML infrastructure (feature stores, orchestration platforms, model serving, annotation tools)

Benefits

Three (3) Medical plan options; Two (2) Dental plan options; Vision plan; Critical Illness, Hospital Indemnity and Accident plans (voluntary); Partially employer funded Health Savings Account; Flexible Savings Accounts (healthcare, dependent care, transportation); Company paid Basic Life and AD&D; Short and Long-Term Disability; Voluntary Life and AD&D plans; Traditional and Roth 401(k) with discretionary employer match; Long-term incentive awards (subject to program guidelines and approvals); Supportlinc wellbeing program; LifeBalance discounts program; Flexible paid vacation; Company-paid holidays (typically nine per year); Paid sick time; Paid parental leave; Identity theft program; Travel assistance; Fitness and other discount programs