Security Data Engineer| 6 Month Contract | (Inside IR35) | Hybrid - 2 days onsite (Ldn or Glasgow) | Starting ASAP
Day Rate: £431
About the Role;
You will design, develop, and implement data solutions on the Azure cloud platform, have a deep understanding of Azure Data Factory, Databricks, and Synapse Analytics, with a proven track record of delivering complex data engineering projects in a banking or financial services environment. Knowledge of security tools and data pipelines a plus.
As a Data engineer you will be integral to the design and implementation of scalable data pipelines that ensures data is collected, stored and processed efficiently.
Work closely with other security and engineering teams, such as data scientists, analysts and business risk to ensure data needs are met, this collaboration leads to:
- Better alignment: Ensuring data design, ingestion, curation aligns to business objectives and supports various departmental needs.
- Enhanced Productivity: Providing data and solutions that empowers teams to perform their roles more effectively along with project outcomes.
Experienced with Azure Data Factory for creating, scheduling and orchestrating ETL workflows, proficient in using Databricks and Spark for data processing and knowledge of Synapse for data integration and analytics.
Focus on improvement utilising tools such as Azure DevOps and a working knowledge of continues integration and continues deployment pipelines (CI/CD).
Expectation of continuous learning, staying up to date with the latest Azure services and features, participate in webinars, online courses and Azure community events.
Main Duties:
- Support the Design and Implementation of Data Architectures: Design and build scalable data pipelines using Azure Data Factory and DataBricks to ingest, transform, and load data from various sources. Develop and optimize data and pipeline architectures within the Azure cloud environment.
- Data Processing and Transformation: Develop and maintain data integration processes to ensure seamless data flow across systems, leveraging Databricks and Synapse Analytics.
- Data Storage Management: Select and manage appropriate data storage solutions like Azure SQL Database, Azure Data Lake, or Azure Blob Storage.
- Data Warehousing: Be able to guide and support the building of scalable data warehousing solutions utilizing Azure Synapse Analytics for handling large datasets.
- Data Modelling and Analysis: Collaborate with data modellers and business analysts to design and implement efficient data models that support business requirements.
- Real-time Data Processing: Configure and optimize streaming pipelines with Azure Stream Analytics for extracting insights from live data streams.
- Data Governance and Security: Ensure data governance practices, implement security measures, and monitor data quality within Azure.
- DevOps and Automation: Automate data pipelines, monitor processes, and manage deployments using tools like Azure DevOps for continuous integration and delivery.
- Performance Optimization: Optimize data processing performance by fine-tuning data pipelines, queries, and storage solutions.
- Data Quality: Implement data quality checks and monitoring processes to ensure data accuracy, consistency, and reliability.
- Collaboration: Work closely with cross-functional teams, including data scientists, software engineers, and product managers, and Senior stakeholders to deliver end-to-end data solutions.
- Documentation: Create and maintain comprehensive documentation of data engineering processes, architectures, and workflows.
- Troubleshooting: Diagnose and resolve data-related issues, ensuring minimal disruption to business operations.
- Continuous Improvement: Stay current with emerging technologies and best practices in data engineering, cloud computing, and the banking industry.
Essential Skills & Experience:
- Proficiency in Azure Data Factory, Databricks, and Synapse.
- Hands-on experience with Python, SQL, and Spark for data processing.
- Strong analytical skills for working with unstructured datasets.
- Ability to design and implement well-written code.
- Experience in automating tasks and deploying production-standard code.
- Excellent communication skills and the ability to collaborate effectively within teams.
- Familiarity with visualization tools and building insightful dashboards.
- Effective communication and collaboration skills with the ability to work in a team-oriented environment.
- Certification in Azure or related technologies (e.g., Microsoft Certified: Azure Data Engineer Associate).
- Excellent organisational, time management skills
- Know of security tools and risk frameworks
- Knowledge of relevant technology standards, roadmaps and best practices
- Knowledge of leading industry standards for retail banking channels and products
- Knowledge of change management principles
- Knowledge of Data Management principles
- Experience with Agile development methodologies.
This role has been deemed Inside IR35 by the client. Applicants must hold, or be happy to apply for, a valid Basic Disclosure Scotland. Please click the link to apply.
