Edge Effects is looking for an experienced, highly analytical Senior Data Engineer to join our core data platform team in London. In this pivotal role, you will lead the architecture, construction, and optimization of our enterprise data infrastructure, enabling real-time analytics, machine learning pipelines, and data-driven product features.
Working alongside Data Scientists, Analytics Engineers, Software Developers, and Cloud Operations teams, you will design scalable cloud data warehouses, automate robust ETL/ELT pipelines, and enforce strict data governance standards. This position blends hands-on pipeline engineering with technical leadership, offering opportunities to shape our long-term data strategy and mentor junior data engineers.
Key Responsibilities
- Data Platform Architecture: Design, build, and maintain high-throughput, cloud-native data platforms (Snowflake, Databricks, BigQuery, or AWS Redshift) to power advanced analytics and reporting.
- Pipeline Engineering (ETL/ELT): Architect, automate, and monitor resilient batch and streaming data pipelines using Python, SQL, Apache Airflow, and dbt (data build tool).
- Data Modeling & Storage: Develop scalable dimensional data models, data lake storage architectures, and schema designs optimized for performance and query cost-efficiency.
- Data Quality & Governance: Enforce data validation frameworks, lineage tracking, role-based access control (RBAC), and strict compliance with UK GDPR and data protection standards.
- CI/CD & DevOps Integration: Integrate data pipeline deployments into automated CI/CD workflows using Docker, Kubernetes, and Terraform (Infrastructure as Code).
- Mentorship & Operational Excellence: Lead technical root-cause analysis for data pipeline failures, establish data engineering best practices, and mentor mid-level data engineers.
Skills & QualificationsEssential
- 5+ years of commercial experience as a Data Engineer or Data Platform Engineer building production-grade data infrastructure.
- Expert-level proficiency in SQL and Python for data processing, manipulation, and pipeline orchestration.
- Hands-on experience with cloud data platforms (Snowflake, Databricks, AWS Redshift, or Google BigQuery).
- Proven background with orchestration tools such as Apache Airflow, Prefect, or Dagster, alongside modern transformation tooling (dbt).
- Strong understanding of data architecture patterns: Data Warehousing, Data Lakes, Data Mesh, and Lambda/Kappa architectures.
- Experience with continuous integration/deployment (CI/CD) and Infrastructure as Code (IaC using Terraform).
- Solid understanding of UK GDPR regulations and data governance protocols.
Desirable
- Relevant Professional Certifications: Snowflake Certified SnowPro Core, Databricks Certified Data Engineer Professional, AWS Certified Data Engineers, or GCP Professional Data Engineer.
- Experience with real-time streaming technologies (Apache Kafka, AWS Kinesis, or Spark Streaming).
- Knowledge of NoSQL databases and vector stores (MongoDB, Redis, Pinecone, or Cassandra).
- Background in supporting ML/AI models (MLOps integration).
Benefits
- Annual performance-based bonus
- Hybrid working model (3 days central London office / 2 days remote)
- 25 days annual leave plus UK bank holidays
- Enhanced pension scheme (up to 8% company match)
- Private medical and dental insurance
- Life assurance (4x base salary)
- £1,500 annual personal learning and professional development budget
- Fully paid professional certifications and exam fees
- Cycle to Work scheme & Electric Vehicle (EV) salary sacrifice scheme
- Enhanced maternity, paternity, and adoption leave
- Regular team socials, tech meetups, and company events
How to Apply
If you are a driven Senior Data Engineer ready to build next-generation data infrastructure at Edge Effects, submit your updated CV today to start the application process.
Pay: £79,000.00-£83,000.00 per year
Benefits:
- Company pension
- Free parking
- Life insurance
- On-site parking
Work Location: In person
