Senior Data Engineer – Quant & AI Infrastructure

About the role

Algocor is building a trading system where research, execution, market data and an LLM‑based agent layer share the same infrastructure. The data layer must be well‑structured, observable, recoverable and trusted by both quant systems and AI agents.

Key responsibilities

• Design, build and operate high‑volume ETL/ELT pipelines from market‑data providers (e.g., Pyth, Databento) and exchange/broker APIs.
• Develop and maintain streaming data pipelines from WebSocket and API sources.
• Own production‑grade time‑series database design and operations using QuestDB, kdb+, ClickHouse, TimescaleDB or similar.
• Define data structures for tick data, OHLCV, symbols, derived signals and internal datasets, handling partitioning, retention, compression and schema evolution.
• Implement robust failure handling for streaming data, including reconnect logic, replay, backfill, duplicate detection and gap detection.
• Write production‑grade Python services and FastAPI endpoints to expose clean, validated data to internal systems and the AI layer.
• Document datasets, schemas, access rules and operational assumptions to enable confident downstream development.

Required profile

• Proven experience building and operating production data systems at high volume.
• Deep understanding of streaming pipeline challenges such as ordering, duplicates, late‑arriving data and recovery.
• Hands‑on experience with time‑series or high‑volume analytical databases (QuestDB, kdb+, ClickHouse, TimescaleDB, etc.).
• Strong Python programming and SQL skills for building maintainable pipelines and APIs.
• Comfort with Docker, Linux, Git, CI/CD, monitoring, alerting and incident response.

Required skills

• Python
• SQL
• Docker
• Linux
• Git
• CI/CD
• Monitoring & alerting
• QuestDB, kdb+, ClickHouse, TimescaleDB
• ETL/ELT pipeline development
• Streaming data (WebSocket, API)