1. Position Summary
The Data Engineering Lead is responsible for designing, building and operating data platforms and pipelines required for AI, analytics and automation use cases. The role leads a team of Data Engineers to implement the data architecture defined by the Architecture Lead and Solution Architect, ensuring reliable, high-quality and well-governed data for use cases such as Manufacturing Control Tower, Golden Batch, Yield Improvement, OOS prediction and pricing models.
2. Roles Played
- Data Engineering team lead for AI and analytics initiatives.
- Owner of data pipelines, transformations and data availability SLAs.
- Key contributor to data modelling, data quality and governance practices.
3. Key Platforms / Technologies
- Azure Data Lake, Data Warehouse, Synapse, Databricks (or equivalent).
- ETL/ELT tools and orchestration frameworks.
- Streaming and batch data ingestion frameworks.
- BI environments (Power BI, Tableau) as data consumers.
- Integration with SAP ECC or S/4 HANA , MES, LIMS, Empower, CTMS and other enterprise sources.
4. Overall Job Responsibilities
A. Data Platform & Pipeline Design (≈ 25%)
- Design data ingestion, transformation and storage solutions as per architecture guidelines.
- Build data models and structures to support AI, analytics and reporting use cases.
- Ensure solutions are modular, reusable and performance optimized.
B. Data Engineering Delivery & Operations (≈ 25%)
- Lead the implementation of data pipelines (batch and streaming) into the data lake/warehouse.
- Set up and maintain orchestration, monitoring, alerting and logging for data flows.
- Own day-to-day data operations and incident resolution related to pipelines.
C. Data Quality, Governance & Security (≈ 20%)
- Implement data quality rules, validations and reconciliations.
- Ensure data lineage and metadata capture for key datasets.
- Work with Security and Governance teams to enforce access control, privacy and data integrity requirements.
D. Performance, Optimization & Cost Management (≈ 10%)
- Optimize pipelines and queries for performance and cost (compute, storage).
- Periodically review resource utilization and recommend improvements.
E. Collaboration & Stakeholder Engagement (≈ 10%)
- Collaborate with Solution Architect, Data Scientists, Full Stack Developers and Business Analysts.
- Understand data needs of each use case and translate them into engineering tasks.
F. Team Leadership & Capability Building (≈ 10%)
- Lead and mentor Data Engineers and ETL developers.
- Promote best practices in coding, testing, documentation and DevOps for data.
5. External Interfaces
- Data platform vendors and implementation partners.
6. Internal Interfaces
- Solution Architect and Architecture Lead.
- BI Developers, Data Scientists and Modelers.
- Application team (SAP, MES, LIMS etc.), Infrastructure and Security.
8. Education
- Bachelor’s degree in engineering / computer science / IT – mandatory.
- Postgraduate qualification in Data Engineering / Data Science – preferred.
9. Experience
- 8–12 years of experience in data engineering, ETL or data warehouse roles.
- Minimum 3–5 years leading data engineering teams or complex data projects.
- Hands-on experience with cloud data platforms (preferably Azure).
- Pharma or manufacturing data experience is an advantage.
10. Knowledge & Skills (Functional / Technical)
- Strong skills in SQL/NoSQL, Spark/Databricks and ETL tools (ADF, Data Build Tool, SSIS – SQL Server Integration Services); streaming (Kafka/Event Hubs)
- Experience with Azure data lake / Storage and Microsoft Fabric Delta Lake, data warehouse architecture
- Familiarity with Python/Scala or other scripting for data pipelines.
- Understanding data quality, governance and metadata concepts.
- Exposure to integration with SAP, MES, LIMS or similar systems.
11. Leadership / Managerial Attributes
- Hands-on technical leader with a delivery focus.
- Good communication and coordination with multiple teams.
- Ability to mentor junior engineers and enforce engineering discipline.
12. Other Requirements
- Relevant cloud/data certifications (e.g., Azure Data Engineer) preferred.
- Willingness to support critical data operations outside normal hours when required.
12-14 years96
Get Personalized Job Matches with 1 Click
Job Description
Download Resume1. Position Summary
The Data Engineering Lead is responsible for designing, building and operating data platforms and pipelines required for AI, analytics and automation use cases. The role leads a team of Data Engineers to implement the data architecture defined by the Architecture Lead and Solution Architect, ensuring reliable, high-quality and well-governed data for use cases such as Manufacturing Control Tower, Golden Batch, Yield Improvement, OOS prediction and pricing models.
2. Roles Played
- Data Engineering team lead for AI and analytics initiatives.
- Owner of data pipelines, transformations and data availability SLAs.
- Key contributor to data modelling, data quality and governance practices.
3. Key Platforms / Technologies
- Azure Data Lake, Data Warehouse, Synapse, Databricks (or equivalent).
- ETL/ELT tools and orchestration frameworks.
- Streaming and batch data ingestion frameworks.
- BI environments (Power BI, Tableau) as data consumers.
- Integration with SAP ECC or S/4 HANA , MES, LIMS, Empower, CTMS and other enterprise sources.
4. Overall Job Responsibilities
A. Data Platform & Pipeline Design (≈ 25%)
- Design data ingestion, transformation and storage solutions as per architecture guidelines.
- Build data models and structures to support AI, analytics and reporting use cases.
- Ensure solutions are modular, reusable and performance optimized.
B. Data Engineering Delivery & Operations (≈ 25%)
- Lead the implementation of data pipelines (batch and streaming) into the data lake/warehouse.
- Set up and maintain orchestration, monitoring, alerting and logging for data flows.
- Own day-to-day data operations and incident resolution related to pipelines.
C. Data Quality, Governance & Security (≈ 20%)
- Implement data quality rules, validations and reconciliations.
- Ensure data lineage and metadata capture for key datasets.
- Work with Security and Governance teams to enforce access control, privacy and data integrity requirements.
D. Performance, Optimization & Cost Management (≈ 10%)
- Optimize pipelines and queries for performance and cost (compute, storage).
- Periodically review resource utilization and recommend improvements.
E. Collaboration & Stakeholder Engagement (≈ 10%)
- Collaborate with Solution Architect, Data Scientists, Full Stack Developers and Business Analysts.
- Understand data needs of each use case and translate them into engineering tasks.
F. Team Leadership & Capability Building (≈ 10%)
- Lead and mentor Data Engineers and ETL developers.
- Promote best practices in coding, testing, documentation and DevOps for data.
5. External Interfaces
- Data platform vendors and implementation partners.
6. Internal Interfaces
- Solution Architect and Architecture Lead.
- BI Developers, Data Scientists and Modelers.
- Application team (SAP, MES, LIMS etc.), Infrastructure and Security.
8. Education
- Bachelor’s degree in engineering / computer science / IT – mandatory.
- Postgraduate qualification in Data Engineering / Data Science – preferred.
9. Experience
- 8–12 years of experience in data engineering, ETL or data warehouse roles.
- Minimum 3–5 years leading data engineering teams or complex data projects.
- Hands-on experience with cloud data platforms (preferably Azure).
- Pharma or manufacturing data experience is an advantage.
10. Knowledge & Skills (Functional / Technical)
- Strong skills in SQL/NoSQL, Spark/Databricks and ETL tools (ADF, Data Build Tool, SSIS – SQL Server Integration Services); streaming (Kafka/Event Hubs)
- Experience with Azure data lake / Storage and Microsoft Fabric Delta Lake, data warehouse architecture
- Familiarity with Python/Scala or other scripting for data pipelines.
- Understanding data quality, governance and metadata concepts.
- Exposure to integration with SAP, MES, LIMS or similar systems.
11. Leadership / Managerial Attributes
- Hands-on technical leader with a delivery focus.
- Good communication and coordination with multiple teams.
- Ability to mentor junior engineers and enforce engineering discipline.
12. Other Requirements
- Relevant cloud/data certifications (e.g., Azure Data Engineer) preferred.
- Willingness to support critical data operations outside normal hours when required.