Role description
Role: Lead Data Engineer
Fixed term contract
Contract Length: Initial 3-6 months with possible extensions
Start Date: ASAP
Experience range - 10- 12 years
Location Requirement: Onsite (3 days per week in the office and 2 days remote )
Applicants must be legally authorized to work in the United Kingdom without the need for current or future visa sponsorship
Responsibilities
- Design, build and optimise robust data ingestion pipelines to acquire, transform and load data vendor platforms into client's data ecosystem.
- Develop scalable data engineering solutions using Databricks, PySpark, SparkSQL and associated cloud technologies.
- Build and maintain secure, reliable and automated data ingestion processes from external vendor systems, including SFTP-based file transfers and other integration methods.
- Ensure data is landed, structured, governed and accessible to support reporting, analytics and business use cases.
- Work with Business Analysts, Product Owners, Architects and delivery squads to translate business requirements into technical data solutions.
- Support data discovery, profiling and validation activities to understand source data structures, data quality issues and data gaps.
- Develop and maintain data transformations, curated datasets and data models required to support reporting and analytical use cases.
- Monitor, troubleshoot and resolve data ingestion issues, defects and enhancements identified during development, testing, UAT and production support.
- Ensure solutions comply with data architecture standards, engineering best practices, security requirements and governance frameworks.
- Produce clear technical documentation for data ingestion processes, data flows and operational support requirements.
- Provide comprehensive handover documentation and knowledge transfer to the Data Support team following delivery of data ingestion pipelines.
- Collaborate within Agile delivery teams, actively contributing to sprint planning, stand-ups, retrospectives and continuous improvement activities.
- Identify opportunities to improve pipeline performance, automation, scalability and maintainability through process and technology enhancements.
- Support knowledge sharing and contribute to Engineering and Data Communities of Practice.
Skills & Experience
Essential
- Proven experience designing, building and supporting enterprise-scale data ingestion pipelines and ETL/ELT solutions.
- Strong hands-on experience with Databricks, PySpark and SparkSQL.
- Experience developing and supporting secure data integrations using SFTP and other file-based or API-driven ingestion mechanisms.
- Experience ingesting and processing structured, semi-structured and unstructured data from internal and third-party source systems.
- Strong understanding of data modelling, transformation techniques and data warehousing principles.
- Experience working with cloud-based data lake and analytics platforms.
- Strong understanding of batch and near real-time data processing patterns.
- Experience conducting data profiling, discovery and validation activities to assess data quality, completeness and suitability for business requirements.
- Experience implementing data quality checks, reconciliations and monitoring processes.
- Ability to investigate and resolve ingestion, transformation and data quality issues identified during testing, UAT or production support.
- Understanding of data governance, security, data lineage and documentation standards.
- Experience producing technical documentation and operational handover materials.
- Strong stakeholder engagement skills with the ability to work effectively across business, architecture, engineering and analytics teams.
- Experience working within Agile delivery environments.
- Knowledge of source control, CI/CD practices and release management processes.
- Ability to work independently while collaborating effectively within cross-functional squads.
Desirable
- Experience integrating data from retail technology platforms, IoT devices or third-party vendor systems.
- Experience working with AI Camera, Computer Vision or Electronic Shelf Edge Label (eSEL) technologies.
- Knowledge of Azure Data Lake, Azure Data Factory and related Azure data services.
- Experience supporting reporting, analytics or BI solutions through the creation of trusted and governed data assets.
- Experience working within large-scale retail or data transformation programmes.
If interested, please apply with your updated CV for an immediate discussion
#UST
Skills
PySpark, Azure Data Factory, Agile, CI/CD
About UST
UST is a global digital transformation solutions provider. For more than 20 years, UST has worked side by side with the world’s best companies to make a real impact through transformation. Powered by technology, inspired by people and led by purpose, UST partners with their clients from design to operation. With deep domain expertise and a future-proof philosophy, UST embeds innovation and agility into their clients’ organizations. With over 30,000 employees in 30 countries, UST builds for boundless impact—touching billions of lives in the process.
