Senior Data Engineer

What You’ll Do

• Design, build, and optimize robust, scalable, and reliable data pipelines to ingest and process data from a wide variety of sources.
• Develop and maintain our data warehousing solutions (e.g. Snowflake), including maintenance of an advanced RBAC model to ensure data governance throughout all data tooling.
• Ensure data quality and integrity by implementing data validation frameworks, monitoring systems, and anomaly detection protocols.
• Architect and manage cloud-based data infrastructure.
• Orchestrate batch machine learning pipelines. Work closely with data scientists to orchestrate code based on data science and product requirements, getting alignment on optimizations when necessary.
• Collaborate with stakeholders, including data scientists, analysts, and product managers to understand data requirements and deliver actionable solutions.
• Improve performance, efficiency and optimize cost of existing data systems and processes.
• Work alongside Data and Engineering leadership to weigh tradeoffs in build vs buy decisions and tradeoffs in product roadmap vs technical initiatives.
• Work with internal software engineering teams to define architectures and best practices for ingesting data into the data warehouse and making aggregated data and metrics from the data warehouse available to our production applications.
• Interact with vendors to craft the adoption of new technology and infrastructure for the team.
• Leverage and drive team adoption of AI coding agents, ensuring best practices are defined and kept up to date as AI tooling advances.
• Take part in an on-call rotation. Work with Engineering leadership to define SLAs and alerting policies for our systems.
• Lead and participate in incident management, ensuring clear and timely communication with stakeholders.
• Be a leader, mentor, and subject matter expert for the team, stakeholders, and peers. Foster a collaborative environment that drives solutions forward at a larger scope.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related field or equivalent professional experience.
• 5+ years experience in data engineering.
• Expert proficiency in SQL and strong programming skills in Python.
• Proven experience building and orchestrating data pipelines using dbt and Airflow
• Hands-on experience developing data solutions in a major cloud environment, with a strong preference for Microsoft Azure.
• Solid understanding of BI-tooling integrations with a cloud data warehouse, with a strong preference for PowerBI
• Solid understanding of event-driven architecture and the use of messaging systems like Kafka to enable real-time data flow.
• Experience with a modern cloud data warehouse like Snowflake, Databricks, BigQuery, or Redshift.
• Familiarity with CI/CD, microservice architecture, domain driven design, data contracts
• Solid understanding with best practices around data security/privacy, role-based access control
• Experience with machine learning concepts, frameworks, and infrastructure
• Excellent written and oral communication and presentation skills