Data Engineer (SC Cleared,Databricks)
other jobs LMA
Added before 2 hours
- England,London,City of London
- Full Time, Permanent
- £70,000 - £75,000 per annum
Job Description:
Full job descriptionSenior Data Engineer (Azure Databricks) - SC Cleared
Location: London, UK (3 days per week onsite)
Salary: Up to £75,000 per annum
Job Type: Permanent, Full-Time
Security Clearance: Active SC Clearance Required
The Opportunity
We are seeking an experienced Senior Data Engineer to join a high-performing team responsible for developing and enhancing a modern Azure-based data platform supporting economic analysis, forecasting, and modelling activities.
This role offers the opportunity to work with large-scale datasets, cloud-native technologies, and modern data engineering practices. You will play a key role in designing and optimising data pipelines, implementing data governance processes, and delivering reliable data solutions that enable critical business insights.
Key Responsibilities
Data Pipeline Development & Optimisation
*Design, develop, and maintain scalable and reliable data pipelines.
*Build ETL/ELT solutions to ingest, transform, and load data from multiple sources, including APIs, databases, and external data providers.
*Optimise pipeline performance, scalability, and cost efficiency.
*Implement data quality controls, validation processes, and monitoring solutions.
Data Transformation & Processing
*Develop complex data transformation logic using Spark (PySpark or Scala).
*Build data cleansing, enrichment, aggregation, and processing solutions.
*Ensure data accuracy, consistency, and reliability throughout the data lifecycle.
Azure Databricks Development
*Develop and support solutions within Azure Databricks.
*Work extensively with Unity Catalog, Delta Lake, Spark SQL, and related Databricks services.
*Apply Databricks best practices for development, deployment, and workload optimisation.
*Develop solutions using SQL, Python, R, YAML, and JavaScript.
Data Integration
*Integrate data from relational databases, APIs, and streaming platforms.
*Design and implement robust data integration patterns.
*Collaborate with development teams to support seamless data exchange across systems.
Data Quality & Governance
*Utilise Azure Purview for data governance and data quality initiatives.
*Implement data lineage, metadata management, monitoring, and alerting capabilities.
*Ensure compliance with organisational data governance standards and best practices.
Collaboration & Delivery
*Work closely with data scientists, analysts, economists, and technical stakeholders.
*Translate business requirements into scalable technical solutions.
*Participate in code reviews, technical discussions, and knowledge-sharing sessions.
*Contribute to Agile delivery processes and continuous improvement initiatives.
DevOps & Automation
*Automate data engineering workflows and deployments.
*Develop and maintain CI/CD pipelines.
*Promote DevOps best practices across the wider engineering team.
Required Skills & Experience
*8+ years of Data Engineering experience (10+ years preferred).
*Minimum 3 years of hands-on Azure Databricks experience.
*Strong expertise in Python, PySpark, and/or Scala.
*Extensive experience with:
*Azure Databricks
*Azure Data Factory
*Azure Blob Storage
*Azure SQL Database
*Delta Lake
*Spark SQL
*Strong understanding of:
*Data architecture
*Data modelling
*Data warehousing
*Data integration patterns
*Experience working with large-scale datasets and complex data pipelines.
*Knowledge of SQL and NoSQL databases.
*Experience with data governance, data quality frameworks, and metadata management.
*Experience using Git and working within Agile delivery environments.
*Excellent communication, stakeholder management, and problem-solving skills.
Desirable Skills
*Azure Purview
*Kafka
*Azure Event Hubs
*Power BI
*Tableau
*Azure DevOps
*Jenkins
*Docker
*Kubernetes
*Experience within Financial Services, Banking, or Economic Data environments
*Microsoft Certified: Azure Data Engineer Associate
*Databricks Certifications
What’s On Offer
*Salary up to £75,000.
*Hybrid working model with 3 days per week in a London office.
*Opportunity to work on a large-scale Azure and Databricks data platform.
*Exposure to enterprise-level data engineering challenges and modern cloud technologies.
*Collaborative and technically focused environment.
*Career development and certification support opportunities.
Please note: Active SC Clearance is required for this position. Applicants without current SC Clearance will not be considered.
Location: London, UK (3 days per week onsite)
Salary: Up to £75,000 per annum
Job Type: Permanent, Full-Time
Security Clearance: Active SC Clearance Required
The Opportunity
We are seeking an experienced Senior Data Engineer to join a high-performing team responsible for developing and enhancing a modern Azure-based data platform supporting economic analysis, forecasting, and modelling activities.
This role offers the opportunity to work with large-scale datasets, cloud-native technologies, and modern data engineering practices. You will play a key role in designing and optimising data pipelines, implementing data governance processes, and delivering reliable data solutions that enable critical business insights.
Key Responsibilities
Data Pipeline Development & Optimisation
*Design, develop, and maintain scalable and reliable data pipelines.
*Build ETL/ELT solutions to ingest, transform, and load data from multiple sources, including APIs, databases, and external data providers.
*Optimise pipeline performance, scalability, and cost efficiency.
*Implement data quality controls, validation processes, and monitoring solutions.
Data Transformation & Processing
*Develop complex data transformation logic using Spark (PySpark or Scala).
*Build data cleansing, enrichment, aggregation, and processing solutions.
*Ensure data accuracy, consistency, and reliability throughout the data lifecycle.
Azure Databricks Development
*Develop and support solutions within Azure Databricks.
*Work extensively with Unity Catalog, Delta Lake, Spark SQL, and related Databricks services.
*Apply Databricks best practices for development, deployment, and workload optimisation.
*Develop solutions using SQL, Python, R, YAML, and JavaScript.
Data Integration
*Integrate data from relational databases, APIs, and streaming platforms.
*Design and implement robust data integration patterns.
*Collaborate with development teams to support seamless data exchange across systems.
Data Quality & Governance
*Utilise Azure Purview for data governance and data quality initiatives.
*Implement data lineage, metadata management, monitoring, and alerting capabilities.
*Ensure compliance with organisational data governance standards and best practices.
Collaboration & Delivery
*Work closely with data scientists, analysts, economists, and technical stakeholders.
*Translate business requirements into scalable technical solutions.
*Participate in code reviews, technical discussions, and knowledge-sharing sessions.
*Contribute to Agile delivery processes and continuous improvement initiatives.
DevOps & Automation
*Automate data engineering workflows and deployments.
*Develop and maintain CI/CD pipelines.
*Promote DevOps best practices across the wider engineering team.
Required Skills & Experience
*8+ years of Data Engineering experience (10+ years preferred).
*Minimum 3 years of hands-on Azure Databricks experience.
*Strong expertise in Python, PySpark, and/or Scala.
*Extensive experience with:
*Azure Databricks
*Azure Data Factory
*Azure Blob Storage
*Azure SQL Database
*Delta Lake
*Spark SQL
*Strong understanding of:
*Data architecture
*Data modelling
*Data warehousing
*Data integration patterns
*Experience working with large-scale datasets and complex data pipelines.
*Knowledge of SQL and NoSQL databases.
*Experience with data governance, data quality frameworks, and metadata management.
*Experience using Git and working within Agile delivery environments.
*Excellent communication, stakeholder management, and problem-solving skills.
Desirable Skills
*Azure Purview
*Kafka
*Azure Event Hubs
*Power BI
*Tableau
*Azure DevOps
*Jenkins
*Docker
*Kubernetes
*Experience within Financial Services, Banking, or Economic Data environments
*Microsoft Certified: Azure Data Engineer Associate
*Databricks Certifications
What’s On Offer
*Salary up to £75,000.
*Hybrid working model with 3 days per week in a London office.
*Opportunity to work on a large-scale Azure and Databricks data platform.
*Exposure to enterprise-level data engineering challenges and modern cloud technologies.
*Collaborative and technically focused environment.
*Career development and certification support opportunities.
Please note: Active SC Clearance is required for this position. Applicants without current SC Clearance will not be considered.
Job number 3938860
Increase your exposure to recruiters with ProJobs
Thousands of recruiters are looking for you in the Job Master profile database, increase your exposure 4 times with a ProJob subscription
You can cancel your subscription at any time.
metapel
Company Details:
LMA
Company size: 50–99 employees
Industry: Recruitment Consultancy
We are LMA Recruitment, a leading recruitment company with over 20 years of experience, connecting talented professionals with top organisations. Foun...