Lead Data Engineer

Lead Data Engineer

ChicagoLondonNew York

Posted Yesterday

With respect to NY, CA, and IL based applicants, the starting base pay range for this role is between USD 175000 and USD 225000 annually. The actual base pay is dependent upon several factors, including, but not limited to, relevant experience, business needs and market demands. This role may also be eligible for bonus compensation and employee benefits.

Role Overview

We are seeking a hands-on Senior / Lead Data Engineer to provide technical leadership for the platforms, pipelines, and data products that power analytics, applications, and investment decision-making across the firm. You will architect scalable, cloud-first data solutions; set engineering standards; and lead small teams through delivery of reliable, analytics-ready datasets and services.

This role combines deep technical execution with mentorship, cross-functional partnership, and ownership of complex data initiatives. It offers the opportunity to take on people-management responsibilities over time.

What You’ll Do

• Lead the design and delivery of scalable ingestion pipelines, data models, and platform services using Python, SQL, Snowflake, and AWS.

• Architect reliable solutions for structured, unstructured, market, and alternative datasets, with particular focus on performance, lineage, usability, and operational resilience.

• Drive the evolution of the Data Acquisition Platform, including APIs, services, plugins, and AI-enabled workflows for onboarding, pipeline creation, metadata generation, and natural-language data access.

• Establish and improve automated data-quality frameworks covering completeness, freshness, schema integrity, reconciliations, and business-rule validation.

• Own technical standards for testing, observability, alerting, incident response, and production support across a large and growing dataset estate.

• Lead root-cause analysis for complex, time-sensitive data incidents and drive durable corrective actions.

• Mentor engineers through design reviews, code reviews, pairing, and technical coaching; help shape team practices and engineering culture.

• Partner directly with Analysts, Quants, Portfolio Managers, and external data providers to translate requirements into robust end-to-end data solutions.

• Evangelize data engineering best practices and influence technical direction across partner teams.

• Potentially manage a small team, including prioritization, delivery planning, feedback, and career development.

What You’ll Bring

• Significant experience building and operating production data platforms, pipelines, and analytics-ready data products.

• Strong Python and SQL skills, with experience across relational and NoSQL data systems.

• Deep experience with Snowflake or comparable modern cloud data warehouses.

• Strong hands-on experience with AWS data and cloud services, including designing secure, scalable, and cost-effective production architectures.

• Experience designing and orchestrating production workflows with Airflow or comparable tools.

• Cloud infrastructure experience in AWS, Azure, or Google Cloud.

• Strong understanding of data modeling, large-scale dataset performance, time-series data, and temporal-query patterns.

• Demonstrated ability to lead technical projects end-to-end, make sound architectural decisions, and improve existing complex systems.

• A track record of mentoring engineers and communicating effectively with both technical and business stakeholders.

Nice to Have

• Experience with Go and service-oriented platform development.

• Experience applying AI/LLM capabilities to data engineering workflows.

• Financial-services or market-data experience.