Data Modelling & Schema Design: Develop and maintain data models (conceptual, logical, and physical) that define how data is stored and related. This includes designing relational schemas, graph data models for knowledge graphs, and time-series data structures as needed, ensuring they accurately represent business entities and relationships. You will continually refine these models to meet AI use cases and evolving business requirements.
Data Storage Architecture: Define and implement data storage and management patterns that optimise data retrieval and analytics performance. This involves selecting or designing appropriate storage solutions (e.g. relational databases, NoSQL/graph databases, data warehouses, data lakes) and structuring them for scalability and fast access to large datasets used in AI projects. Ensure the architecture can handle structured and unstructured data and is cloud-ready for elasticity.
Data Pipelines & Integration: Build and oversee robust data pipelines (ETL/ELT processes) to integrate data from multiple sources into centralised platforms. You will design workflows to collect, transform, and load data into analytics repositories or feature stores, guaranteeing that AI models have consistent, well-prepared data to work with. This includes setting up stream processing for real-time data when required and automating pipeline orchestration for efficiency.
Data Governance & Quality: Establish and enforce data governance policies and standards. This means defining practices for data quality, data cleaning, and master data management, as well as setting security and privacy controls to protect sensitive information. You will ensure compliance with relevant data regulations and implement data security measures (e.g. access controls, encryption) and validation rules so that the data used in AI is trustworthy and compliant.
Metadata Management & Lineage: Implement frameworks for data metadata management and lineage tracking. This includes maintaining data catalogues or dictionaries that describe data meaning (possibly leveraging ontologies), and tools or processes to trace how data flows through pipelines and transformations. By providing transparency into data origins and transformations, you support model interpretability and enable troubleshooting of data issues, which is critical in AI development.
Collaboration with Engineering Teams: Work closely with data engineers, ML engineers, and data scientists to ensure the data architecture meets their needs. You will collaborate on designing data interfaces (e.g. APIs or query endpoints) and assist in shaping how data is used for features in machine learning. This role requires translating requirements between data teams and ML teams, and jointly resolving issues to streamline the path from raw data to AI insights.
Performance Optimisation & Scaling: Monitor the performance and scalability of the data infrastructure, and tune it as the AI project grows. Optimise database queries, indexing, and storage layouts for faster model training and inference data access. Plan for scale by leveraging cloud capabilities (compute, storage) and manage costs effectively, adjusting architectures (partitioning, caching, etc.) to maintain efficient, cost-effective operations as data volumes increase. You may also evaluate new technologies (e.g. distributed computing frameworks or new databases) and incorporate them to continually improve the architecture.
