Data Engineering Consultant

Responsibilities

Define and maintain the enterprise data architecture blueprint, including data lakes, warehouses, streaming platforms, and analytics layers; Design logical, physical, and conceptual data models to support operational, analytical, and AI use cases; Ensure architecture supports scalability, performance, and high availability; Architect & Build Data Pipelines: Design, construct, install, test, and maintain highly scalable data management systems and ETL/ELT pipelines; Integrate Diverse Data Sources: Develop processes to ingest and integrate high-volume, high-velocity data from SCADA systems, historians (like OSIsoft PI, Aspen InfoPlus.21), DCS, PLC, and IoT sensors; Cloud Data Platform Development: Implement and manage data solutions on the Microsoft Azure cloud platform, Leveraging services like Azure IoT Hub, Azure Event Hubs, and Azure Stream Analytics for real-time ingestion and processing of operational technology (OT) data; Data Modelling & Warehousing: Design and implement data models optimized for time-series data from industrial assets, supporting operational dashboards and real-time analytics; Enable Advanced AI: Build the data infrastructure to support AI/ML models for predictive maintenance, operational anomaly detection, and process optimization using real-time OT data

Requirements

10+ years of experience in data architecture, data engineering, or enterprise analytics roles; Strong experience designing enterprise-scale data platforms in cloud and hybrid environments; Expertise in data modelling (conceptual, logical, physical) and data integration patterns; Deep understanding of data lakes, data warehouses, Lakehouse architectures, and real-time streaming platforms; Hands-on experience with cloud data services (Azure preferred); Strong knowledge of SQL, data transformation frameworks, and metadata management; Familiarity with DevOps and DataOps practices; Strong communication skills with the ability to engage both technical and business stakeholders; Expert-level proficiency in SQL and Python for data manipulation and pipeline development; Hands-on experience with distributed computing frameworks like Apache Spark (PySpark); Deep experience with Microsoft Azure (Azure Data Lake Storage, Azure Data Factory, Azure Databricks, Azure Synapse); Proven experience with modern data platforms Databricks Delta Lake; Understanding of machine learning lifecycles and the data requirements for training and deploying AI/ML models; Proficiency with Git and CI/CD best practices

Nice to have

Certifications such as Azure Data Engineer Expert, Azure Solutions Architect, or equivalent; Hands-on experience with the OSDU™ Data Platform; Experience in Oil & Gas, energy, or industrial data environments; Understanding of cybersecurity considerations for OT environments and data segregation; Experience integrating data from ERP systems like SAP; Experience working within AI-first or analytics-driven transformation programs; Advanced SQL skills, including query optimization and performance tuning; Experience of constructing and maintaining enterprise knowledge graphs; Experience with streaming technologies like Kafka is a plus