Digital Platforms, Senior Data Engineer

The Senior Data Engineer (Databricks Platform) is a senior individual contributor responsible for the hands-on design, development, optimization, and operational support of the enterprise data platform built on Databricks and AWS technologies. This role ensures scalable, secure, and high-performance data pipelines, models, and integrations that power analytics, reporting, and advanced AI/ML use cases across the organization. The role brings deep expertise in distributed data processing, Delta Lake architecture, data modeling, ETL/ELT frameworks, and cloud environments (AWS). This role focuses on execution excellence, platform reliability, and technical problem resolution, while collaborating closely with Data Platform Lead, Product, Project Management, Testing, Customer Success, and external development partners.

Responsibilities

Design, develop, and maintain scalable data pipelines using Databricks (Spark / Delta Lake); Build robust ETL/ELT workflows leveraging PySpark and Databricks SQL; Implement and support medallion architecture (Bronze, Silver, Gold layers); Design and maintain efficient data models, including 3NF, star schema, and canonical models; Support and maintain Data Catalog and Data Governance frameworks; Build and maintain CI/CD pipelines; Collaborate with external partners for development activities and ongoing production support; Demonstrate hands-on experience with AWS Glue, S3, and Lambda; Provide operational support for AWS-hosted infrastructure; Monitor cloud environments for availability, performance, security, and cost efficiency; Support deployment activities and ensure environment readiness; Partner with Testing and QA teams to support UAT and validation activities; Collaborate closely with Product Managers, Project Managers, QA, DevOps, and external development partners; Translate business and operational requirements into scalable technical solutions; Support secure and reliable integration with internal enterprise systems and data sources; Ensure adherence to PCI, cGMP, GCP, and regulatory policies

Requirements

Distributed data processing; Delta Lake architecture; Data modeling; ETL/ELT frameworks; Cloud environments (AWS); Databricks (Spark / Delta Lake); PySpark and Databricks SQL; Medallion architecture (Bronze, Silver, Gold layers); 3NF, star schema, and canonical models; Data Catalog and Data Governance frameworks; CI/CD pipelines; AWS Glue, S3, and Lambda; Operational support for AWS-hosted infrastructure; Cloud monitoring for availability, performance, security, and cost efficiency