Senior Data Engineer

Luxoft -
London

Apply Now

Job details

1 day ago

Qualifications

Data development experience
CI/CD
Data integration
Cloud architecture
Data quality analysis
Encryption
English
Data and database experience
ETL data integration
CI/CD pipelines
Scala
Terraform
Data warehousing
S3
Kafka
Design documentation
Redshift
Leadership
Data warehouse
Infrastructure provisioning automation
Terraform provisioning
Identity & access management

Full job description

Project description

This is a strategic data engineering engagement with our client to architect and plan the migration of their entire data processing and ETL estate from Matillion to AWS Glue — a foundational shift in how one of the world's largest financial market infrastructure companies handles its data pipelines.
During this Mobilisation Phase, you'll work jointly with client's engineering teams to reverse-engineer the existing landscape, design the target-state architecture across every layer (infrastructure, data processing, workflows, dependencies, and operating model), and build the detailed delivery blueprint that will greenlight the full-scale migration.
The work is technically rich and highly collaborative: you'll review and validate job inventories spanning hundreds of ETL workflows, define reusable migration patterns and templates, design a validation and reconciliation framework, run a proof-of-concept to stress-test the approach, and navigate client's rigorous internal governance — from Architectural Significance Assessments through Architectural Review Boards to a formal Gate 1 decision.
This is the kind of engagement where your recommendations directly shape a multi-phase, multi-million-pound programme: the target-state framework you produce here becomes the blueprint that a larger delivery team will execute against. Perfect opportunity for combining deep data engineering knowledge with architecture leadership, stakeholder influence, and structured delivery planning inside a Tier 1 financial services environment

Responsibilities

Design, build, and maintain ETL/ELT pipelines on AWS

Migrate existing data pipelines and workloads from legacy/on-prem systems to AWS

Develop and optimize data models for data warehouse/data lake (Redshift, S3)

Build and orchestrate data workflows (Glue, Step Functions, Lambda, EMR)

Implement data quality checks, validation, and reconciliation processes

Optimize pipeline performance and manage compute/storage costs

Ensure data security and access control (IAM, KMS, encryption)

Monitor and troubleshoot pipeline failures (CloudWatch, logging)

Collaborate with data analysts, BI developers, and architects on data requirements

Document data pipelines, architecture, and operational runbooks

Skills

Must have

5+ years of experience

Hands-on experience building data pipelines on AWS (Glue, EMR, Lambda, Step Functions)

Strong SQL and experience with data warehousing (Redshift, dimensional modeling)

Proficiency in Python or Scala for data engineering

Experience with S3-based data lake architecture (partitioning, cataloging, formats like Parquet)

Experience migrating data pipelines from on-prem or other cloud platforms

Understanding of data governance, security, and access control on AWS

Experience with CI/CD for data pipelines

Nice to have

AWS certification (Data Analytics Specialty or Data Engineer Associate)

Experience with streaming data (Kinesis, MSK/Kafka)

Familiarity with Infrastructure as Code (Terraform/CloudFormation)

Experience with orchestration tools (Airflow, dbt)

Experience in financial domain

Other

Languages

English: C2 Proficient

Seniority

Senior

London, United Kingdom of Great Britain and Northern Ireland

Req. VR-124691

Strategy & Design Consulting (SI)

BCM Industry

04/09/2026

Req. VR-124691

Apply Now

Project description

Responsibilities

Skills

Other

Jobseeker tools

Employer Tools

Browse

Stay Connected