The client is a private equity firm
Basic information:
Location: 100% remote
Rate: 170-215 PLN/hour net + VAT
Employment type: B2B contract
Duration: 12 months + extensions
Recruitment process: 2 stages
English: B2+/C1
Luxmed, MultiSport, equipment provided by the client
About the Role
We are hiring a Senior Data Engineer & Team Lead with deep Databricks expertise to join our Accelerator Development team. You will be a hands-on technical lead responsible for designing, building, and owning end-to-end data solutions on our Azure Databricks platform.
This is an ownership role. You will not just write code - you will define approach, set standards, drive delivery from requirements through to production, and actively mentor colleagues. You will be the subject matter expert for Databricks across the team.
You will also be responsible for the team around you, facilitating communication of work, regular check-ins, aiding & developing skills of team, advocating where necessary to stakeholders, working to help through problems / blockers, ensuring adherence to best practice and calling out alterations to practice where appropriate. You will have a strong ownership of team dynamics & development.
Key Responsibilities
- Own end-to-end delivery of data engineering solutions: from scoping and design through build, testing, deployment, and ongoing support.
- Architect and build scalable data pipelines and workflows in Databricks - Jobs, Notebooks, Spark Declarative Pipelines (SDP), and Workflows.
- Implement Change Data Capture (CDC) and Change Data Feed (CDF) patterns using native Databricks capabilities to enable near-real-time and incremental data processing.
- Design and manage Unity Catalog resources: define data assets, apply access controls, provision permissions for users and service principals, and enforce data governance policies.
- Select and configure the appropriate triggering mechanism for each use case (scheduled, file-arrival, continuous, API-driven) to ensure reliable pipeline orchestration.
- Promote solutions through environments using CI/CD pipelines and Declarative Automation Bundles (DAB), maintaining clean branching strategies and automated deployments via GitHub Actions or Azure DevOps.
- Develop Delta Lake and Lakehouse patterns (Bronze / Silver / Gold medallion architecture) aligned to our existing data platform.
- Collaborate with BAs and business stakeholders to translate requirements into well-scoped technical solutions.
- Define and enforce engineering standards: code quality, testing (unit, integration), documentation, and reusable component libraries.
- Mentor and upskill team members on Databricks capabilities, data engineering best practices, and platform tooling.
- Support and resolve production issues, including performance tuning of Spark workloads and cluster configuration.
- Engage the QA function early in the delivery lifecycle - agree acceptance criteria and data quality expectations during design, not at handover.
- Build solutions with testability in mind: modular notebooks, parameterised jobs, and clearly defined input/output contracts that QA can validate independently.
- Treat QA sign-off as part of the definition of done - a pipeline is not complete when it runs; it is complete when it has passed agreed checks and data quality assertions are in place.
- Shared responsibility with Scrum Master to align work with team members, overseeing the delivery of stories and features within sprints.
- Ownership of best practice implementation for the team on SDLC & coding standards.
- Check-ins with team to ensure effective engagement & team coherence
Databricks - Core (Essential)
You must be able to demonstrate hands-on production experience with all of the following:
- Building and maintaining Databricks Workflows and Jobs (multi-task, with dependencies, retries, and alerting).
- Writing production-quality PySpark and Python notebooks; applying modular, testable design patterns.
- Spark Declarative Pipelines (SDP): building declarative pipelines, using expectations for data quality, and understanding pipeline modes (triggered vs. continuous).
- Change Data Feed (CDF) and Change Data Capture (CDC): enabling CDF on Delta tables, processing change streams, and applying CDC patterns for downstream consumers.
- Unity Catalog: creating and managing catalogs, schemas, and tables; granting and revoking privileges; understanding the megastore hierarchy; provisioning access for users, groups, and service principals.
- Triggering options: configuring schedule-based, file-arrival (Auto Loader), continuous-mode, and API/webhook-driven triggers; understanding trade-offs between each.
- CI/CD with Declarative Automation Bundles (DAB): defining bundle configurations, managing environments (dev/int/uat/prod), and deploying via automated pipelines.
- Source control: feature branching, pull requests, code review, and integration with GitHub or Azure DevOps.
Azure & Data Platform
- Azure Data Lake Storage Gen2 (ADLS) - storage account hierarchy, ABAC/RBAC, mounting and external locations.
- Azure Key Vault integration for secrets management within Databricks.
- Azure SQL / T-SQL - querying, stored procedures, and performance optimisation.
- Medallion architecture (Bronze / Silver / Gold) and Lakehouse design principles.
- Python (data engineering): libraries including pyspark, delta, pytest, and common data manipulation tooling.
Engineering Practices
- CI/CD pipelines - GitHub Actions or Azure DevOps Pipelines; automated testing, linting, and deployment gates.
- Infrastructure as Code (IaC) - Pulumi or Terraform for provisioning Azure resources.
- Test-driven development (TDD) - unit and integration tests for pipeline logic.
- Software design patterns - clean, reusable, maintainable code; understanding of SOLID principles.
- Agile delivery - sprint planning, backlog grooming, estimation, and retrospectives.
- AI-assisted development - actively uses AI tools throughout the development lifecycle: to analyse existing solutions and codebases (understanding logic, identifying patterns, spotting edge cases), to accelerate the build of new solutions (transformation logic, test generation, boilerplate, code review), and to generate and
validate documentation. Treats AI as a first-class part of the engineering workflow, not an occasional shortcut.
Non-Technical
- Proven experience (2+ years) of leadership in practice (team lead, lead developer, other).
- Ownership mindset - comfortable taking full responsibility for a deliverable, raising blockers early, and driving to completion without
hand-holding.
- Stakeholder communication - ability to explain technical decisions to non-technical audiences clearly and concisely.
- Mentoring - track record of upskilling colleagues and conducting meaningful code reviews.
- Analytical rigour - able to translate ambiguous business needs into precise technical requirements.
- Organised and communicative with QA - raises upcoming work in sprint planning, avoids last-minute test requests, explains transformation logic and edge cases clearly, and responds to defects without friction.
- AI pragmatism - understands where AI tooling genuinely accelerates delivery and where it introduces risk; applies it as a tool, not a crutch, and can articulate that distinction clearly to the team. Actively thinks about where AI use cases could be introduced into data workflows and is able to propose, scope, and validate those ideas rather than waiting to be directed.
Hays Poland sp. z o.o. is an employment agency registered in a registry kept by Marshal of the Mazowieckie Voivodeship under the number 361.
Niemcy
Wielka Brytania
Holandia
Irlandia
Norwegia
Szwajcaria