Senior Data Engineer (Python, PySpark, Kafka)

Develop and maintain robust Python code for data manipulation, scripting, and application development. Design and build high-performance data transformation jobs using PySpark on large-scale datasets. Build and maintain real-time data streaming pipelines using Apache Kafka, including Kafka Connect and Kafka Streams implementations. Work with the Hadoop ecosystem, including HDFS, YARN, Hive, and related technologies. Extract, query, and manage data using SQL and relational databases (e.g., PostgreSQL, SQL Server). Apply data warehousing concepts and dimensional modeling principles to optimize data architecture. Manage code repositories and collaborate with teams using version control systems (e.g., Git, GitHub).

Responsibilities

Develop and maintain robust Python code for data manipulation, scripting, and application development; Design and build high-performance data transformation jobs using PySpark on large-scale datasets; Build and maintain real-time data streaming pipelines using Apache Kafka, including Kafka Connect and Kafka Streams implementations; Work with the Hadoop ecosystem, including HDFS, YARN, Hive, and related technologies; Extract, query, and manage data using SQL and relational databases (e.g., PostgreSQL, SQL Server); Apply data warehousing concepts and dimensional modeling principles to optimize data architecture; Manage code repositories and collaborate with teams using version control systems (e.g., Git, GitHub)

Requirements

B.Tech; M.Tech; 6 to 9 Years; Python; PySpark; Apache Kafka; Kafka Connect; Kafka Streams; Hadoop ecosystem (HDFS, YARN, Hive); SQL; relational databases (PostgreSQL, SQL Server); data warehousing concepts; dimensional modeling principles; version control systems (Git, GitHub)