Job Summary
job description
DATA ENGINEER
JOB SUMMARY
--------------------------------------------------------------------------------
An individual contributor writing production Python that moves, transforms, and
serves data across the business. This is a software engineering role: the work is
designing, coding, testing, and operating Python services, libraries, and
pipelines — not configuring tools or hand-writing queries. Builds systems that
are readable, tested, observable, and cheap to change, working with analytics,
engineering, and business teams to deliver reliable data products.
PRINCIPAL ACCOUNTABILITIES
--------------------------------------------------------------------------------
Engineering & Solution Delivery (70%)
- Design, build, and maintain batch and streaming pipelines in Python,
including the orchestration, scheduling, and monitoring around them.
- Write reusable, well-typed Python libraries and internal packages that other
engineers and analysts build on.
- Build Python services and APIs that expose data to downstream applications,
and integrate with third-party and internal APIs.
- Write unit, integration, and data-contract tests, and keep pipelines covered
by automated CI.
- Profile and optimize Python code and data processing jobs for runtime,
memory, and cost.
- Deploy and operate code in the cloud using containers, infrastructure as
code, and CI/CD.
- Enforce access controls, secrets handling, and sensitive-data protections at
every stage of the data lifecycle.
Continuous Improvement (30%)
- Find and fix efficiency, reliability, cost, and correctness problems in
existing code and pipelines, and refactor toward simpler designs.
- Replace one-off scripts and notebooks with tested, packaged, scheduled code.
- Implement data quality checks, validation, and monitoring that catch issues
before consumers do.
KNOWLEDGE, SKILLS, AND ABILITIES
--------------------------------------------------------------------------------
- Strong, idiomatic Python: data structures, typing, error handling,
generators/iterators, context managers, and the standard library.
- Software engineering fundamentals: modular design, dependency management,
packaging, semantic versioning, and API design.
- Testing discipline: pytest, fixtures, mocking, and writing code that is
testable by construction.
- Debugging and profiling; able to reason about performance, concurrency, and
memory in Python.
- Working knowledge of SQL, data modeling, governance, and privacy practices
sufficient to design sound schemas and query them effectively.
- Able to explain technical trade-offs to both technical and business
audiences.
EXPERIENCE
--------------------------------------------------------------------------------
- Shipping production Python, not just scripts or notebooks.
- Building and operating ETL/ELT pipelines at scale.
- Python data tooling: pandas, Polars, PyArrow, DuckDB, or PySpark.
- Schema enforcement and validation: Pydantic or similar.
- Workflow orchestration in code: Airflow, Dagster, Prefect, dbt, or Temporal.
- Testing and quality tooling: pytest, pyright, ty, pyrefly, ruff or black.
- Dependency and environment management: uv, Poetry, pip-tools, or conda.
- At least one public cloud (AWS, Azure, or GCP) and its Python SDK, with
familiarity with the others.
- Cloud data platforms: Snowflake, Databricks, BigQuery, or Redshift.
- Relational and non-relational databases, including schema design.
Preferred
- Python web/API frameworks (FastAPI, Flask).
- Async Python, multiprocessing, or other concurrency patterns.
- Streaming: Kafka, Kinesis, or Spark Structured Streaming.
- Infrastructure as code and automation (Terraform, GitHub Actions).
- Docker and Kubernetes.
- A second language: Go, Rust, Scala, or TypeScript.
- Generative AI, machine learning, or data science tooling.
EDUCATION
--------------------------------------------------------------------------------
- Bachelor's degree, or an equivalent combination of education and experience.
RECOMMENDED YEARS OF RELEVANT EXPERIENCE
--------------------------------------------------------------------------------
- 3-5+ years
Pay: £400.00-£500.00 per day
Work Location: Remote
