Full job descriptionESSENTIAL ROLES & RESPONSIBILITIES
*Identify and understand customer data-centric use cases within regulated financial services environments
*Design and implement data ingestion, processing, and transformation pipelines on Azure
*Build and maintain data pipelines for cleaning, normalisation, enrichment, and preparation
*Apply appropriate data modelling techniques and architecture patterns, with a strong focus on medallion architecture
*Orchestrate, monitor, and optimise Azure Databricks jobs and Azure Data Factory pipelines across development, UAT, and production environments
*Configure platforms, clusters, and compute resources to optimise performance, cost, and reliability
*Use automated CI/CD pipelines to manage, deploy, and version data artefacts and pipelines
*Operationalise workflows developed by analysts and data scientists
*Support customers in adopting Azure data, analytics, and machine learning services
*Ensure secure storage, processing, and quality of customer data
*Ensure networking and security best practices are applied when designing and operating data solutions
*Design solutions for processing large volumes of data using batch and streaming approaches
*Collaborate with analytics teams on data visualisation best practices and reporting enablement
*Ensure all solutions are well-documented, including pipelines, schemas, transformations, and operational runbooks
GOVERNANCE & REPORTING
*Maintain accurate documentation of data pipelines, schemas, transformations, and deployment processes
*Support data governance initiatives including lineage, metadata management, and access control
*Contribute to service reporting, risk tracking, and continuous improvement actions
*Ensure data environments are audit-ready and aligned with governance standards
TECHNOLOGY STACK (AZURE)
Cloud Platform:
*Microsoft Azure
Data Engineering & Analytics:
*Azure Databricks (development, UAT, and production)
*Azure Data Factory
*Azure Synapse Analytics (where applicable)
Machine Learning & AI:
*Azure Machine Learning (limited non-production usage)
*Azure Document Intelligence
Databases:
*Microsoft SQL Server / Azure SQL Database (primary platforms)
*PostgreSQL (limited use)
*MySQL (limited use)
Data Processing:
*Batch and streaming data pipelines
Security & Governance:
*Role-based access control (RBAC)
*Data encryption and key management
*Audit logging and monitoring
DevOps:
*CI/CD pipelines for data artefacts and infrastructure
BEHAVIOURAL COMPETENCIES – ORGANISATIONAL & BEHAVIOURAL FIT
*Positive mindset and enthusiasm for learning new technologies
*Collaborative and supportive team player
*Strong sense of ownership and accountability
*Methodical, analytical approach to problem-solving
*Strong understanding of ethical data usage in regulated environments
CRITICAL COMPETENCIES – TECHNICAL FIT
Essential:
*Strong SQL skills
*Programming experience with Python and/or Scala
*Hands-on experience with Azure-based data platforms
*Experience designing, building, and maintaining data pipelines
*Strong understanding of data modelling (relational and analytical), including medallion architecture
*Experience orchestrating and optimising Databricks and Data Factory workloads
*Experience using CI/CD pipelines for data and analytics solutions
*Strong awareness of security, networking best practices, GDPR, and PII handling
Desirable:
*Experience with Azure Databricks in production environments
*Familiarity with Azure Machine Learning and AI services
*Exposure to data visualisation tools (e.g. Power BI)
*Experience with big data frameworks (Spark, Kafka)
*Knowledge of data governance, lineage, and metadata tooling
SHIFT & WORKING PATTERN
*Standard business hours, with participation in an on-call rota as required
*Occasional weekend engineering coverage will be required, typically limited to a small number of planned weekends per year to support business continuity, resilience testing, or disaster recovery activities