Data Engineer - Informatica/ADF/Fabric/Python/SQL

The Data Engineer will independently design, build, and optimize cloud-based data pipelines supporting analytics and reporting initiatives. Responsibilities include selecting ingestion patterns, engineering high-performance processing jobs, analyzing execution plans, and optimizing compute resources, partition strategies, latency, and cloud costs. The role will implement governable Lakehouse and Medallion data layers using Microsoft Fabric, Databricks, Azure Data Factory, and Azure Data Lake Storage. Additional duties include developing advanced Python and PySpark transformation modules, managing pipelines through code and version control, supporting CI/CD workflows, and diagnosing network, data-distribution, and code-performance issues. The engineer will apply strong SQL skills, establish scalable engineering standards, and mentor junior team members.

Responsibilities

Independently design, build, and optimize cloud-based data pipelines supporting analytics and reporting initiatives; Selecting ingestion patterns; Engineering high-performance processing jobs; Analyzing execution plans; Optimizing compute resources, partition strategies, latency, and cloud costs; Implementing governable Lakehouse and Medallion data layers using Microsoft Fabric, Databricks, Azure Data Factory, and Azure Data Lake Storage; Developing advanced Python and PySpark transformation modules; Managing pipelines through code and version control; Supporting CI/CD workflows; Diagnosing network, data-distribution, and code-performance issues; Applying strong SQL skills; Establishing scalable engineering standards; Mentoring junior team members

Requirements

Bachelor’s degree in Computer Science, Information Systems, Engineering, a related technical field, or equivalent experience; 8+ years of experience in data engineering, software engineering, or big data development; Experience independently architecting and building end-to-end data pipelines in a cloud environment; Strong proficiency in Python, PySpark, and SQL; Experience with Microsoft Fabric, Databricks, Azure Data Factory, and Azure Data Lake Storage; Experience with Lakehouse, Medallion, and Lambda or Kappa architectures; Experience with CI/CD, Azure DevOps, Git, and orchestration tools; Ability to optimize distributed data-processing jobs, compute sizing, partitioning, and cloud costs

Nice to have

Experience developing infrastructure and data pipelines as code; Knowledge of Spark internals and distributed-computing concepts; Experience with Delta and Parquet file formats; Experience diagnosing systemic pipeline, network, data-skew, and code-performance issues; Ability to mentor junior and associate data engineers; Ability to travel as required

Benefits

Weekly Direct Deposit; 401K Matching; Competitive medical, dental and vision insurance; Consistent communication throughout your project; ZeekTek Referral Program