Principal Application Support Engineering – AI Products & Agents

As a Principal Engineer – Operations, Reliability & AI Platforms, you will provide technical leadership for enterprise-scale production systems, cloud platforms, and AI-powered applications. You will drive reliability, operational excellence, observability, automation, and resiliency across multiple products and platforms while serving as a trusted engineering leader for critical business services. This role combines deep technical expertise, architectural leadership, operational ownership, and strategic thinking. You will partner with Engineering, Product, Platform, Security, and Operations teams to establish reliability standards, eliminate operational toil through automation, improve system resilience, and ensure mission-critical services operate securely and efficiently at scale.

Responsibilities

Define and drive the operational engineering strategy for enterprise applications, cloud-native platforms, and AI-enabled systems; Establish standards for reliability, scalability, availability, performance, security, and operational excellence; Lead architecture reviews with a focus on supportability, observability, resiliency, disaster recovery, and long-term sustainability; Provide technical leadership during major incidents, service disruptions, and high-severity production events; Influence product and engineering roadmaps to proactively address technical debt, operational risks, and reliability improvements; Own reliability outcomes through SLOs, SLIs, error budgets, and operational health metrics; Drive adoption of Site Reliability Engineering (SRE) and operational excellence practices across teams; Lead complex root cause investigations involving distributed systems, cloud platforms, integrations, databases, and AI services; Establish operational readiness reviews, production acceptance criteria, release governance, and supportability standards; Define and implement resiliency engineering practices including failover validation, disaster recovery testing, capacity planning, and business continuity programs; Drive continuous improvement through incident trend analysis, reliability metrics, and operational insights; Provide technical leadership for AI-powered applications, copilots, virtual assistants, chatbots, and agent-based systems operating in production environments; Establish operational standards for LLMs, RAG, AI agents, vector databases, MCP ecosystems, and orchestration platforms; Lead investigations involving AI model failures, retrieval issues, prompt orchestration, tool-calling workflows, agent interactions, and integration failures; Define monitoring strategies for AI quality, token consumption, latency, throughput, reliability, and operational health; Drive AI governance, responsible AI practices, security, compliance, and operational risk management; Lead initiatives to eliminate manual operational effort through automation, intelligent workflows, self-healing capabilities, and autonomous remediation; Design and implement operational automation using scripting, Infrastructure as Code, CI/CD pipelines, observability platforms, and AIOps solutions; Establish enterprise runbooks, playbooks, operational standards, and knowledge management frameworks; Drive adoption of platform engineering principles to improve reliability, developer productivity, and operational efficiency; Partner with Engineering, Product, Platform, Security, Compliance, and Operations teams to improve service quality and customer outcomes; Mentor engineers and technical leaders in reliability engineering, distributed systems, cloud operations, and AI platform support; Communicate technical risks, operational health, and strategic recommendations to both technical and executive stakeholders; Lead cross-functional initiatives that improve reliability, resilience, supportability, and operational maturity across the organisation

Requirements

10+ years of experience in Software Engineering, Site Reliability Engineering, DevOps, Platform Engineering, Cloud Engineering, Production Operations, or related disciplines; Proven experience operating and supporting large-scale enterprise production environments; Experience leading reliability, operational excellence, and platform transformation initiatives; Demonstrated success driving operational improvements across multiple products, systems, and teams; Experience operating within regulated industries such as Life Sciences, Healthcare, Financial Services, or similar environments; Deep expertise in distributed systems, microservices, APIs, event-driven architectures, and cloud-native platforms; Strong understanding of software architecture, performance engineering, resiliency, security, and operational excellence; Advanced troubleshooting capabilities across applications, integrations, databases, networking, infrastructure, and cloud services; Experience leading major incident management, post-incident reviews, root cause analysis, and corrective action programmes; Strong understanding of governance, compliance, risk management, and secure engineering practices; Expert knowledge of AWS, Azure, and/or GCP cloud platforms; Experience designing and supporting highly available, fault-tolerant, enterprise-scale cloud architectures; Strong expertise with Kubernetes, Docker, and container orchestration technologies; Experience with Infrastructure as Code technologies such as Terraform, CloudFormation, Bicep, or similar platforms; Deep understanding of networking, identity management, security, and cloud governance; Experience implementing platform engineering principles and Internal Developer Platforms (IDPs); Strong knowledge of high availability, failover, disaster recovery, backup, and resilience engineering concepts; Expert understanding of SRE principles, reliability engineering, error budgets, SLOs, and SLIs; Strong experience with GitHub Actions, Azure DevOps, Jenkins, GitLab CI/CD, and modern deployment automation platforms; Proven experience implementing self-healing systems, automated remediation, and operational automation solutions; Deep expertise with observability platforms including OpenTelemetry, Grafana, Datadog, Splunk, ELK, Prometheus, Dynatrace, or equivalent technologies; Experience implementing AIOps, intelligent alerting, event correlation, and operational analytics platforms; Strong automation and scripting capabilities using Python, PowerShell, Bash, JavaScript, or similar technologies; Strong understanding of Generative AI ecosystems and enterprise AI architecture patterns; Experience supporting LLM-based Applications, Retrieval-Augmented Generation (RAG), AI Agents and Multi-Agent Systems, Agentic Workflows, Semantic Search Platforms, Vector Databases, and AI Orchestration Frameworks; Familiarity with Azure OpenAI, AWS Bedrock, Google Vertex AI, OpenAI APIs, Anthropic Claude, Azure AI Services, or similar AI platforms; Understanding of MCP Server architecture, MCP Clients, Tool Calling, Enterprise AI integrations, and AI platform operations; Experience monitoring AI reliability, latency, throughput, token consumption, usage patterns, and operational health metrics; Knowledge of Responsible AI, AI governance, security, compliance, model evaluation, and operational risk management; Availability to work flexible work hours; Candidate should be open to work on different shifts - 6AM to 2PM and 2PM to 11PM when required

Nice to have

Experience leading enterprise-wide operational transformation initiatives; Experience building reliability engineering or operational excellence programmes; Experience supporting Kubernetes-based cloud-native architectures at scale; Familiarity with LangChain, LangGraph, Semantic Kernel, AutoGen, CrewAI, OpenAI Agents SDK, or similar orchestration frameworks; Experience supporting MCP-based enterprise integrations and AI ecosystems; Experience implementing AIOps capabilities, autonomous operations, and intelligent automation solutions; Cloud, Kubernetes, DevOps, SRE, or AI/Generative AI certifications; Experience supporting mission-critical, global, 24x7 production environments