Databricks Data Operations Specialist

Scicom Infrastructure Services is seeking a Databricks Data Operations Specialist to support the operational stability, reliability, performance, and security of a large-scale Databricks data platform. This role will work as part of a multidisciplinary Databricks team that includes data engineers, architects, business analysts, governance specialists, and technical managers. The Data Operations Specialist will be responsible for monitoring production data pipelines, responding to incidents, troubleshooting platform and data-processing issues, coordinating deployments, and ensuring that data products meet established availability, quality, and service-level requirements.

Responsibilities

Monitor the health, availability, performance, and capacity of Databricks workspaces, jobs, pipelines, clusters, SQL warehouses, and related cloud services; Provide day-to-day operational support for production Databricks environments; Monitor scheduled and event-driven data pipelines for failures, delays, data-quality issues, and performance degradation; Troubleshoot Databricks jobs, Spark workloads, cluster failures, notebook errors, connectivity issues, and data-processing exceptions; Restart, rerun, recover, or coordinate remediation of failed workflows according to documented procedures; Perform controlled data reprocessing, backfills, reconciliation, and recovery activities; Validate source-to-target record counts, control totals, schema consistency, and business-rule compliance; Serve as first or second level of support for production Databricks and data-pipeline incidents; Triage incidents and coordinate incident response; Conduct or support root-cause analyses and document incident timelines and corrective actions; Develop and maintain dashboards, alerts, and operational reports; Monitor compliance with established data contracts and execute data-quality controls; Support release and deployment of Databricks notebooks, jobs, workflows, libraries, and configurations across environments; Support Databricks identity, access, and entitlement administration, including Unity Catalog permissions; Monitor Databricks and cloud-resource consumption and recommend cost and performance optimizations; Create and maintain operational procedures, troubleshooting guides, and support runbooks

Requirements

Bachelor's degree in computer science, information technology, data engineering, information systems, or a related field, or equivalent professional experience; At least four years of experience in data operations, production support, data engineering, cloud operations, DevOps, platform support, or a related technical role; At least two years of hands-on experience supporting Databricks environments or production Databricks workloads; Experience supporting production ETL, ELT, batch, or streaming data pipelines; Working knowledge of Databricks, Apache Spark, PySpark or Python, SQL, Delta Lake, Databricks Workflows or Jobs, cloud object storage, data pipeline monitoring, and log analysis and troubleshooting; Experience investigating failed jobs, delayed pipelines, schema issues, data discrepancies, and performance problems; Familiarity with medallion architecture and Bronze, Silver, and Gold data layers; Experience with at least one major cloud platform: Microsoft Azure, AWS, or Google Cloud; Experience using ticketing and service-management tools such as ServiceNow, Jira, or a comparable platform; Experience with monitoring, alerting, logging, or observability tools; Understanding of incident, problem, change, and release-management processes; Ability to write and maintain technical procedures and operational documentation; Strong analytical, organizational, communication, and troubleshooting skills; Ability to work effectively with engineers, business analysts, architects, security teams, and nontechnical stakeholders; Ability to support multiple production priorities in a structured and timely manner

Nice to have

Databricks Certified Data Engineer Associate or Professional; Databricks Certified Associate Developer for Apache Spark; Microsoft Azure, AWS, or Google Cloud certification; Experience with Unity Catalog, Delta Live Tables, Lakeflow, Auto Loader, Structured Streaming, or Databricks SQL; Experience with Azure Data Factory, Azure Data Lake Storage, Azure Monitor, Log Analytics, AWS Glue, Amazon S3, CloudWatch, or comparable cloud services; Experience with Terraform, Git, Azure DevOps, GitHub Actions, Jenkins, or other CI/CD tools; Experience with Splunk, Datadog, Grafana, Prometheus, or similar observability platforms; Experience supporting APIs, Kafka, Event Hubs, Kinesis, Airflow, dbt, Snowflake, or Power BI; Knowledge of data contracts, schema registries, metadata management, data lineage, and data-quality frameworks; Familiarity with OCDS, procurement data, contracting data, financial data, or public-sector data standards; Experience supporting government, regulated, or high-security environments; Experience operating systems with formal service-level agreements and 24-hour support requirements; Experience with disaster recovery, backup, archival, and operational-resilience testing