Lead Data Software Engineer – Spark, Databricks, Kafka, Kubernetes

About the role

We are looking for a Lead Data Software Engineer to own the migration of an existing analytics platform to a cloud‑agnostic, on‑premises capable solution. You will guide a team in building robust streaming and batch pipelines while ensuring high code quality and rapid delivery.

Key responsibilities

• Implement streaming and batch Spark pipelines running on Kubernetes.
• Develop components of the data generator framework.
• Deploy and test solutions in local and development environments.
• Enhance and refine the evolving data platform.
• Investigate and resolve bugs, and conduct unit testing.
• Participate in refinement, planning, and demo sessions.
• Guide and mentor team members while coordinating with the team lead.

Required profile

• 5+ years of experience with Apache Spark, Databricks, and Kubernetes.
• Proficiency in Python.
• Experience with Confluent/Apache Kafka.
• Completed EPAM Data Engineering course or equivalent qualification.
• Ability to quickly adapt and dive into active implementation phases.
• Strong communication skills, proactive attitude, and ability to work independently while collaborating with a lead.
• English proficiency at B2 level or higher.

Required skills

• Apache Spark
• Databricks
• Kubernetes (K8S)
• Python
• Confluent Apache Kafka

What we offer

• Continuous upskilling, learning & development programs.
• Personal development plan, mentoring, and leadership development.
• Certification support and access to over 2,500 internal courses.
• Competitive compensation paid in USD with regular salary and performance reviews.
• Private health insurance and well‑being events.
• Recreation areas, kitchen, snacks, sports equipment, and IT equipment.
• Extra leave days and referral bonuses.