The Senior Data Engineer is responsible for designing, developing, and managing data solutions on the Data Platform. Building robust and scalable data pipelines, managing data storage and processing systems, and ensuring data quality and integrity.

Responsibilities

Data Pipeline Development:
-- Design, develop, and maintain complex data pipelines to efficiently extract, transform, and load (ETL) data from various sources into target data systems.
-- Master’s the usage of technologies such as Azure Data Factory, Azure Databricks, or SQL Server Integration Services (SSIS) to handle data ingestion, transformation, and integration tasks.

Data Storage and Management:
-- Implement and manage data storage solutions using Microsoft technologies like Azure SQL Database, Azure Data Lake Storage, or SQL Server.
-- Ensure data integrity, security, and compliance with data governance policies.

Data Modeling and Architecture:
-- Responsible for design and implement large data models to support efficient data storage and retrieval.
-- Develop database schemas, define relationships, and optimize performance based on the specific requirements of the data solution.
-- Provide advice and participate in the architectural decision-making.

Data Quality and Governance:
-- Guide and implement data quality checks and processes to ensure data accuracy, consistency, and completeness.
-- Collaborate with data stewards and business stakeholders to define and enforce data governance policies.

Big Data and Analytics:
-- Lead the processing and analysis of big data technologies such as Azure HDInsight, Azure Databricks, or Apache Spark..
-- Develop scalable and performant data solutions for analytics, reporting, and machine learning purposes.

Data Security and Compliance:
-- Implement security measures to protect sensitive data and ensure compliance with relevant regulations and standards.
-- Define and enforce access controls, data encryption, and data retention policies.

Performance Tuning and Optimization:
-- Optimize data solutions for performance and scalability.
-- Identify and resolve performance bottlenecks, optimize SQL queries, and fine-tune data processing workflows.

Collaboration and Documentation:
-- Work collaboratively with cross-functional teams, including data scientists, data analysts, and business stakeholders.
-- Document data engineering processes, system architecture, and data flow diagrams for knowledge sharing and future reference.

Collaboration and Stakeholder Management:
-- Collaborate with business stakeholders, data analysts, data architect, data scientists across the organisation to understand their data needs and align data architecture with organizational goals.
-- Guide cross-functional teams on data engineering.
-- Facilitate communication between technical and non-technical stakeholders to ensure effective data-driven decision-making.

Education Required:
-- Bachelor’s or Master’s degree in Computer Science, Information Systems, or a related field

Experience Required:
-A minimum of seven years of progressively responsible postgraduate experiencein data management with a focus on Microsoft Dynamics / F&O Suite data model
-- Proven track record as a Data Engineer or similar role.
-- Experience in Data Lake and Data Lakehouse implementation (e.g., Microsoft data lake, MSQL Server, MySQL).
-- Experience in Data Vault implementation
-- Proficiency in programming languages such as Python, SQL, or Java. Experience with big data technologies like Apache Hadoop, Spark, or Flink.
-- Experience in data governance, architecture and handling large datasets and data pipelines
-- Familiarity with data security, and data privacy regulations (e.g., GDPR, CCPA).

Knowledge, Skills and Language

-- Excellent problem-solving and analytical skills.
-- Ability to develop strategy, and proven skills in translating strategy in roadmaps, workstreams, setting ambitious and meaningful targets
-- Strong knowledge of data modeling techniques, including conceptual, logical, and physical data modeling.
-- Proficiency in database design, data warehousing, and data integration concepts.
-- Understanding of data analytics and business intelligence concepts and tools.
-- Knowledge in Data Mesh, Microsoft Azure Synapse and Azure Data Factory
-- Fluently spoken and written English.