Data Quality Engineer

Job Description

Design and run the data quality layer of our new Databricks lakehouse using Databricks Labs DQX. You will turn business rules from the legacy SQL platform into automated checks, quarantine bad records before they reach Gold tables, and give the business visibility into data health.

Responsibilities

Profile source and Bronze data with the DQX profiler and turn the output into reviewed quality rules.; Define DQX checks (completeness, uniqueness, validity, ranges, referential and custom SQL checks) in YAML/JSON config or code.; Set criticality per rule (error vs warn) and implement split/quarantine patterns for invalid rows.; Embed DQX checks into Databricks Jobs and Lakeflow (DLT) pipelines for batch and streaming data.; Store check definitions and results in Delta tables under Unity Catalog, build DQ dashboards and alerts.; Translate legacy SQL validation logic (constraints, stored procedure checks) into equivalent DQX rules.; Define DQ thresholds that act as quality gates for layer promotion (Bronze → Silver → Gold).; Version-control rules and deploy them through CI/CD.; DQX checks live on every Silver and Gold table in scope.; Invalid records quarantined with a documented triage process.; ADQ dashboard the business uses to track trends and SLA breaches.

Requirements

3+ years in data engineering or data quality roles.; Hands-on PySpark and Spark SQL on Databricks.; Practical experience with DQX, or strong experience with a comparable framework (Great Expectations, Soda, Deequ, DLT expectations) and willingness to ramp up on DQX quickly.; Delta Lake and Unity Catalog fundamentals.; Understanding of data quality dimensions and how to measure them.; Git-based workflow and CI/CD basics.

Nice to have

Azure Data Factory orchestration of Databricks jobs.; Databricks SQL dashboards, Lakehouse Monitoring or Power BI for DQ reporting.; Experience with SQL Server to cloud migrations.; Databricks Certified Data Engineer Associate/Professional.