Principal Software Engineer (Data Engineering)

We are seeking a highly skilled **Principal Software Engineer – Data Engineering** to design, develop, and optimize scalable data platforms and pipelines. The ideal candidate will have strong expertise in **Python, SQL, Snowflake (or equivalent cloud data warehouse), cloud technologies, CI/CD practices, and AI-enabled data solutions**. This role requires technical leadership, architecture ownership, mentoring capabilities, and hands-on engineering excellence.

Key Responsibilities

10+ years of experience in Data Engineering.

DevOps & CI/CD

- Experience with: Azure DevOps, GitHub Actions, Jenkins, GitLab CI/CD, Knowledge of Docker and Kubernetes., Infrastructure as Code experience.

AI & Modern Data Technologies

Exposure to

- Machine Learning pipelines
- Generative AI concepts
- LLM integrations
- Vector databases
- RAG architectures
- Prompt engineering basics

Preferred Qualifications

Success Metrics

- Reliable and scalable data platform delivery.
- Reduced pipeline failures and improved SLA compliance.
- Improved data quality and governance.
- Faster deployment cycles through CI/CD automation.
- Successful adoption of AI-enabled data solutions.
- Increased engineering productivity and platform efficiency.

Responsibilities

Design and implement scalable, reliable, and secure data pipelines.; Develop ETL/ELT solutions for processing large-scale structured and unstructured datasets.; Build data ingestion frameworks supporting batch and near real-time processing.; Design and optimize data models for analytics, reporting, and machine learning workloads.; Ensure high data quality, governance, observability, and reliability across data platforms.; Architect and maintain cloud-native data solutions on AWS or Azure or GCP.; Develop and optimize solutions using **Snowflake** or equivalent cloud data warehouse technologies.; Implement data lake and lakehouse architectures.; Optimize storage costs, compute utilization, and query performance.; Develop production-grade applications and frameworks using Python.; Perform code reviews and establish engineering best practices.; Drive architectural decisions and technology selection.; Ensure adherence to security, scalability, and reliability standards.; Design and implement CI/CD pipelines for data and application deployments.; Automate build, test, deployment, and monitoring processes.; Manage Infrastructure as Code (IaC) using Terraform, CloudFormation, or similar tools.; Improve deployment reliability and observability.; Enable AI/ML workflows by building robust feature and training data pipelines.; Work with AI teams to operationalize machine learning models.; Explore Generative AI, LLM integrations, vector databases, and AI-driven automation solutions.; Support model monitoring, governance, and deployment pipelines.; Provide technical leadership across multiple engineering teams.; Mentor senior engineers and establish engineering standards.; Collaborate with architects, product managers, and business stakeholders.; Drive innovation and strategic technology initiatives.

Requirements

Strong expertise in **Python programming**.; Advanced SQL development and performance tuning skills.; Experience with **Snowflake** or equivalent cloud data warehouse technologies.; Strong understanding of data modeling, ETL/ELT frameworks, and data governance.; Experience with cloud platforms:; Microsoft Azure (Preferred); AWS; Google Cloud Platform

Nice to have

Experience building enterprise-scale data platforms.; Knowledge of Spark, Databricks, Kafka, or streaming technologies.; Understanding of Data Mesh and modern data architecture principles.; Experience in observability and monitoring tools.; Snowflake, Azure, AWS, or GCP certifications are a plus.; Technical leadership; Solution architecture; Strategic thinking; Stakeholder management; Problem-solving; Mentoring and coaching; Strong communication skills; Agile development practices

Benefits

Flexible working environment; Volunteer time off; LinkedIn Learning; Employee-Assistance-Program (EAP)