Principal SRE

We are seeking a Principal Site Reliability Engineer (SRE) to help ensure the stability, scalability, and reliability of our production systems. The ideal candidate will have extensive experience collaborating in cross functional teams to manage the design, implementation and ongoing management of the Backblaze core infrastructure. You will be approved to demonstrate strong observability skills, process and service improvement and ability to leverage AI and automation to assist to mitigate issues faster and meet customer demand. As a technical leader in Backblaze you will be expected to work across the organisation to achieve our goals. This will include working with our executives and customers. The role will focus long term strategic focus on platform stability.

Responsibilities

Own the availability and durability of critical services across production environments and accountability for outcomes; Work independently, take ownership, and drive projects from problem discovery through resolution; Assist and advise on the design of service health metrics using SLIs, SLOs, and error budgets, and escalate issues when thresholds are at risk; Partner with service owners to define/iterate SLIs/SLOs and establish error budget policies and alerting; Participate in leadership on-call rotations, incident response, and post-incident reviews; Follow established ITIL/OSS processes (incident, change, problem, and capacity management); Design automation for common operational tasks, reducing manual intervention and toil; Collaborate with engineering on overall storage design and performance expectations against product SLAs; Define the monitoring, logging, and alerting frameworks (e.g., Prometheus, Grafana, Catchpoint, ELK) and help integrate runbooks with FireHydrant; Work with CI/CD pipelines, configuration management, and infrastructure as code tools (Terraform, Ansible, Jenkins); Write scripts (Bash, Python, Go, etc.) to improve system reliability and efficiency; Partner with engineering, customer success, product, and operations teams to support resilient system design and operations; Work with customers to ensure stability prioritization is based around their needs; Work with vendors and service providers to troubleshoot service issues and track SLA performance; Document systems, share learnings, and help grow a reliability-minded engineering culture; Lead cross functional teams to identify gaps in observability and operations process and define overall prioritization strategy; Identify recurring issues, propose long-term improvements, and design action plans to implement them; Promote reliability-focused practices within development and operations teams

Requirements

Bachelor's degree in Computer Science, Engineering, or related field (or equivalent experience); 10+ years of experience in site reliability, systems engineering, or operations; Exposure to large-scale, production-grade systems, including experience establishing operational procedures or runbooks and bringing others up to speed on them; Solid Linux systems administration and troubleshooting skills; Proven ability to leverage AI tools to improve platform stability and scalability; Knowledge of storage systems, performance, sharding and storage algorithms; Experience in load balancing services including IPVS; Security-minded operations (patching, hardening, identifying vulnerabilities); Familiarity with service reliability concepts (monitoring, alerting, incident response, and root cause analysis); Proficiency in at least one scripting language (Python, Bash, or Go); Experience working with containers (Kubernetes, Docker) and microservices concepts; Comfortable operating in a Kubernetes/Vitess-on-K8s environment (kubectl, mysqlsh, Vitess keyspaces); Experience with Hashicorp products (Nomad, Vault, Terraform)

Nice to have

Experience in a SaaS, service provider, or distributed systems environment; Familiarity with ITIL/OSS practices and SLO/SLA's; Strong problem-solving skills and willingness to learn new technologies; Experience with cloud platforms (AWS, GCP, or Azure); Experience with hyperscalers

Benefits

Healthcare for family, including dental and vision; Competitive compensation and 401K; RSU grants for full-time employees; ESPP program; Flexible vacation policy; Maternity & paternity leave; MacBook Pro plus a generous stipend to personalize your workstation; Childcare bonus (human children only); Fertility treatment and support; Learning & development program; Commuter benefits; Culture that supports a healthy work-life balance