Azure Data Engineer

We are looking for a Azure Data engineer to work on a finance reconciliation project.

Job Duties:

Designing and developing data pipelines within a delta Lakehouse Architecture

Contribute to the design and development of the data models used for analytics and reporting

Reconciliation of on-prem SQL Server data to Azure Cloud

Experience:

• Extensive experience in designing, developing, and managing end-to-end data pipelines, ETL (Extract, Transform, Load), and ELT (Extract, Load, Transform) solutions.
• Maintains a proactive approach to staying updated with emerging technologies and a strong desire to continuously learn and adapt to new tools, languages and methodologies in the ever-evolving field of data engineering and analytics.
• Possesses expert knowledge of the contemporary data engineering and analytics landscape, including current tools, pertinent approaches and likely challenges.
• A strong history of designing and constructing databases, analytics systems and Business Intelligence (BI) platforms within the context of Big Data.
• A demonstrable ability to mentor and coach junior data engineers, sharing knowledge and expertise to help them grow in their roles.
• The capability to collaborate efficiently with business stakeholders to understand their requirements and translate them into effective data solutions.
• Strong support for Continuous Integration/Continuous Deployment (CI/CD) practices and an understanding of agile working methodologies.
• Knowledge and experience working with Delta Lake, a storage layer that brings ACID transactions to data lakes.
• Understanding of the Azure Platform set up including Subscriptions, Resource Group, IAM, Networking and DevOps.
• Experience in setting up processes for cost/performance monitoring Spark jobs to improve maintainability.

Technical Skills Required:

• Azure (ADF, Functions, Blob Storage, Data Lake Storage, Logic Apps, DevOps)
• Databricks
• Spark
• Delta Lake
• SQL
• Python
• PySpark
• ADLS

Education:

Bachelor’s degree in Computer Science or a related field.

Salary offered: £30,000 to £45,000 per annum