Senior Data Platform Engineer

About the role

We are seeking a Senior Data Platform Engineer to manage and scale our core data infrastructure while empowering engineering and Data Science teams with reliable, streamlined processes. This is a long‑term contractor position with overlap hours until 11 am PST.

Key responsibilities

• Manage, maintain, and scale on‑premise Kubernetes clusters, Airflow orchestrations, and distributed Spark infrastructure.
• Build and maintain robust CI/CD pipelines to streamline deployment, data governance, and authentication protocols.
• Perform routine file compaction, storage optimization, and maintenance across HDFS and cloud object stores.
• Maintain system observability, monitor health, and participate in an on‑call rotation using Prometheus, Grafana, and OpenTelemetry.
• Apply core data engineering principles to optimize table formats (Iceberg/Hive), partitioning, schema design, and query performance.

Required profile

• Proven background in distributed systems with deep expertise in managing Kubernetes resources in production.
• Hands‑on experience scaling and managing infrastructure for heavy Apache Spark workloads.
• Strong proficiency in modern CI/CD pipelines, Docker containerization, and data pipeline orchestration (Python/Airflow).
• Deep understanding of data engineering fundamentals, including schema design, partitioning, type safety, and modern table formats (Iceberg/Hive/Trino/Presto).
• Practical experience implementing authentication, access control, and data governance frameworks.
• Solid experience with observability, monitoring, and on‑call practices using Prometheus, Grafana, and OpenTelemetry.

Required skills

• Kubernetes
• Apache Spark
• Airflow
• CI/CD pipelines
• Docker
• Python
• Iceberg
• Hive
• Trino
• Presto
• Prometheus
• Grafana
• OpenTelemetry

What we offer

• Opportunity to work on bleeding‑edge projects
• Work with a highly motivated and dedicated team
• Competitive salary
• Flexible schedule
• Professional development opportunities