Senior Data Infrastructure Engineer – Web Scraping & Scale

About the role

We are looking for a Senior Data Infrastructure Engineer to design and scale high‑performance web‑scraping and data‑ingestion systems. In this fully remote position you will build the core infrastructure that collects millions of social‑media profiles, posts and video records each day, tackling complex anti‑bot challenges and ensuring reliable data delivery at scale.

Key responsibilities

• Lead development and maintenance of scalable data‑collection pipelines for platforms such as Instagram, TikTok, YouTube and X.
• Design and manage web‑scraping infrastructure using residential proxies, TLS fingerprinting, browser automation and session rotation.
• Implement distributed queues and workflow orchestration with Temporal, Ray, Apache Kafka and Celery.
• Architect storage solutions using Parquet, Apache Iceberg, Snowflake and Amazon S3.
• Set up monitoring, automated alerts and recovery mechanisms to handle platform changes, blocking patterns and proxy failures.

Required profile

• At least 4 years of experience in high‑volume web scraping, data engineering or reverse engineering.
• Strong expertise in overcoming advanced anti‑bot systems (Cloudflare, Akamai, PerimeterX) using TLS impersonation, browser automation and proxy management.
• Proven ability to design fault‑tolerant, high‑throughput systems and troubleshoot complex ingestion challenges.

Required skills

• Advanced proficiency in Python or Go.
• Experience with Playwright, Puppeteer, Scrapy or Selenium.
• Knowledge of Temporal, Ray, Apache Kafka, Redis.
• SQL expertise and familiarity with modern data lakes/warehouses (Snowflake, Amazon S3, Parquet, Apache Iceberg).

What we offer

• Fully remote work arrangement.
• Full‑time permanent employment.