Data Engineer
Working in a Vibrant and Exciting Financial environment, the Data Engineer will be responsible for the development and delivery of performant data pipelines withing the firm’s data platform delivering data from diverse sources for use in Business Intelligence and Machine Learning solutions.
Responsibilities:
* Designing and implementing data pipelines feeding data from source systems for Business Intelligence and Machine Learning solutions
* Monitoring and supporting the existing data pipelines.
* Design and evolution of the Data Platform
* Advanced data modelling skills
* Data transformation within the firm’s data platform for Business Intelligence and Machine Learning solutions
* Proficient in performance analysis and tuning
* Ensure data security policies are applied appropriately to the data platform
* Assist with database administration tasks and support hours appropriate to your level
Competencies:
* A collaborative team player, approachable, self-efficient and influences a positive work environment
* Demonstrates curiosity
* Resilient in a challenging, fast-paced environment
* Excels at building relationships, networking and influencing others
* Strategic collaborator with insight and agility, able to anticipate future challenges, ensuring operational effectiveness
Skills and Experience:
Essential
* Experience of the Azure Data Platform including Data Factory, Data Lake, Azure Databricks, Synapse Analytics
* Advanced SQL skills (SQL, T-SQL, PL/SQL)
* Spark (Databricks)
* Scripting skills (Python, PowerShell)
* Data Lake Design - optimal layout for performance, security etc
* Experience in managing and migrating large volumes of data. Compression, partitioning and ETL/ELT
* Understanding of both batch and stream processing.
Desirable
* Experience of other data cloud technologies - AWS, Azure, Oracle Cloud.
* Power BI
* MongoDB
* Splunk
* Data platform automation tools (Azure Devops, Redgate Flyway, Liquibase)
* Experience of working in an Agile/SCRUM environment
Soft
* Able to communicate effectively with varied colleagues ranging from senior stakeholders in different business areas through to the technical implementation teams.
* Think long-term, but deliver iteratively, and show value in small but frequent advancements.
* Understand industry trends and best practice and be selective about what is applicable the organisation.