Senior Data Engineer - Azure Databricks/ SC Clearance
other jobs Crimson
Added before 3 Days
- England,London,City of London
- Full Time, Contract
- £600 - £620 per day
Job Description:
Senior Data Engineer - Azure Databricks/ SC Clearance - Contract
Active SC Clearance is required for this position
Hybrid working - 3 days / week on site required
Up to £620 / day - Inside IR35
We are currently recruiting for a well experienced Senior Data Engineer, required for a leading global transformation consultancy, based in London. The Senior Data Engineer will be responsible for providing hands-on, technical expertise within an agile team. The ideal candidate will be well experienced in building and optimising data pipelines, implementing data transformations, and ensuring data quality and reliability. This role requires a strong understanding of data engineering principles, big data technologies, cloud computing (specifically Azure), and experience working with large datasets
Key skills and responsibilities:
*Design, build, and maintain scalable ETL pipelines for ingesting, transforming, and loading data from APIs, databases, and financial data sources into Azure Databricks. Optimize pipelines for performance, reliability, and cost, incorporating data quality checks.
*Develop complex transformations and processing logic using Spark (PySpark/Scala) for cleaning, enrichment, and aggregation, ensuring accuracy and consistency across the data lifecycle.
*Work extensively with Unity Catalog, Delta Lake, Spark SQL, and related services. Apply best practices for development, deployment, and workload optimization. Program in SQL, Python, R, YAML, and JavaScript.
*Integrate data from relational databases, APIs, and streaming sources using best-practice patterns. Collaborate with API developers for seamless data exchange.
*Utilize Azure Purview for governance and quality monitoring. Implement lineage tracking, metadata management, and compliance with governance standards.
*Partner with data scientists, economists, and technical teams to translate requirements into solutions. Communicate effectively with technical and non-technical audiences. Participate in code reviews and knowledge-sharing sessions.
*Automate pipeline deployments and implement CI/CD processes in collaboration with DevOps teams. Promote best practices for automation and environment management.
Interested? Please submit your updated CV to Lewis Rushton at Crimson for immediate consideration.
Not interested? Do you know someone who might be a perfect fit for this role? Refer a friend and earn £250 worth of vouchers!
Active SC Clearance is required for this position
Hybrid working - 3 days / week on site required
Up to £620 / day - Inside IR35
We are currently recruiting for a well experienced Senior Data Engineer, required for a leading global transformation consultancy, based in London. The Senior Data Engineer will be responsible for providing hands-on, technical expertise within an agile team. The ideal candidate will be well experienced in building and optimising data pipelines, implementing data transformations, and ensuring data quality and reliability. This role requires a strong understanding of data engineering principles, big data technologies, cloud computing (specifically Azure), and experience working with large datasets
Key skills and responsibilities:
*Design, build, and maintain scalable ETL pipelines for ingesting, transforming, and loading data from APIs, databases, and financial data sources into Azure Databricks. Optimize pipelines for performance, reliability, and cost, incorporating data quality checks.
*Develop complex transformations and processing logic using Spark (PySpark/Scala) for cleaning, enrichment, and aggregation, ensuring accuracy and consistency across the data lifecycle.
*Work extensively with Unity Catalog, Delta Lake, Spark SQL, and related services. Apply best practices for development, deployment, and workload optimization. Program in SQL, Python, R, YAML, and JavaScript.
*Integrate data from relational databases, APIs, and streaming sources using best-practice patterns. Collaborate with API developers for seamless data exchange.
*Utilize Azure Purview for governance and quality monitoring. Implement lineage tracking, metadata management, and compliance with governance standards.
*Partner with data scientists, economists, and technical teams to translate requirements into solutions. Communicate effectively with technical and non-technical audiences. Participate in code reviews and knowledge-sharing sessions.
*Automate pipeline deployments and implement CI/CD processes in collaboration with DevOps teams. Promote best practices for automation and environment management.
Interested? Please submit your updated CV to Lewis Rushton at Crimson for immediate consideration.
Not interested? Do you know someone who might be a perfect fit for this role? Refer a friend and earn £250 worth of vouchers!
Job number 3133258
Increase your exposure to recruiters with ProJobs
Thousands of recruiters are looking for you in the Job Master profile database, increase your exposure 4 times with a ProJob subscription
You can cancel your subscription at any time.
metapel
Company Details:
Crimson
Company size:
Industry:
Crimson Education was founded in 2013 with a vision to equalize the university admissions playing field, serving as the launching pad that equips stud...