Data Engineer – Databricks & Azure

About the role

Surgo Health is building the Behavioral Intelligence layer for healthcare, turning conversations into structured data that helps organizations understand and anticipate human behaviour. It is looking for a Data Engineer to join the R&D Engineering team behind Derin, its AI-powered interview platform that runs LLM-led interviews over WhatsApp/SMS, web and voice. You will own the data layer: pipelines moving interview data into a Databricks lakehouse, the models that make it analysis-ready, and the serving layer (Postgres/Lakebase) read in real time.

Key responsibilities

• Use AI coding assistants (Claude Code, Cursor and others) to design, build, debug and optimise code
• Own the Databricks platform, from data ingestion to analytics-ready tables
• Design and govern batch and incremental pipelines on Databricks (Lakeflow Spark Declarative Pipelines, Jobs, Delta Lake) in Unity Catalog
• Migrate the production data layer to Databricks Lakebase (managed Postgres), including sync, migration and cutover
• Make pipelines observable with data-quality checks, tests, lineage, monitoring and alerting
• Turn scientists' notebooks into reusable production pipelines
• Apply privacy and security controls (Unity Catalog access controls, row filters, column masks) in a SOC2 and HIPAA-compliant environment
• Deploy workloads with Terraform and Databricks Asset Bundles via CI/CD

Required profile

• At least 3 years of data engineering experience building production pipelines in Python and SQL
• Hands-on production experience with Databricks (Spark, Delta Lake, Unity Catalog, Lakeflow or Jobs)
• University degree in Computer Science or a related field
• Fluent English, written and spoken, and comfortable working remotely with substantial overlap with US Eastern and UK hours
• Independent, proactive and able to deliver end-to-end with minimal supervision

Required skills

• Relational databases, especially PostgreSQL (schema design, query optimisation, replication or CDC)
• Microsoft Azure (required)
• Infrastructure as code and CI/CD
• Data modelling for analytics and BI, with attention to data quality and testing
• Serving data to LLM-powered features (text-to-SQL, RAG, metric views, semantic layers) is a strong plus
• Survey, research or healthcare data, and regulated data (HIPAA, GDPR/KVKK), is a plus

What we offer

• Competitive compensation, with possible annual performance-based bonuses and equity
• Professional development opportunities
• A public benefit corporation with an impact and equity mission