Top Skills:
Data QualityInformation TechnologyData ExtractionData ManagementAnalyticsData ModelingMicrosoftPerformance ManagementData ProcessingBusiness IntelligenceArchitectureAzureAzure Data FactoryAzure DevopsBatch ProcessingCi/cdData ArchitectureData ExtractionData GovernanceData IntegrationData LakeData ModelingData PipelineData ProcessingData QualityData RetrievalData SecurityDatabricksDevopsEtlFutureIndexingPysparkSparkSqlUnstructured Data
Job Title: Azure Data Engineer
Experience: 3 to 12 Years
Location: Bangalore, PAN India
Employment Type: Full-Time, Hybrid
Key Responsibilities:
1) Design and Development:
- Design, develop, and deploy scalable data pipelines using Databricks (PySpark, Spark SQL) , Azure Data Factory , and other Azure data services.
- Implement ETL/ELT processes to ingest, transform, and load data from various sources into data lakes and data warehouses.
- Optimize and tune data pipelines for performance and scalability.
2) Data Processing:
- Write and optimize complex SQL queries for data extraction, transformation, and analysis.
- Use PySpark for large-scale data processing and analytics.
- Implement data partitioning, bucketing, and indexing strategies for efficient data retrieval.
3) Data Integration:
- Integrate data from multiple sources, including structured, semi-structured, and unstructured data.
- Work with APIs, streaming data, and batch processing to ensure seamless data integration.
4) Data Governance and Quality:
- Implement data governance practices to ensure data quality, consistency, and security.
- Monitor and troubleshoot data pipelines to ensure data accuracy and availability.
5) Collaboration:
- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
- Work closely with DevOps teams to deploy and monitor data pipelines in production environments.
6) Documentation:
- Document data pipelines, workflows, and processes for knowledge sharing and future reference.
- Maintain up-to-date documentation on data architecture and data models.
Must-Have Skills:
- Databricks: Handson experience with Databricks for data processing, analytics, and Serverless SQLWH, Unity Catalog, Lakehouse, Medallion Architecture
- Azure Data Services: Proficiency in Azure Data Factory, Azure Data Lake Storage, and Azure SQL Database, Key Vault, Azure Pricing Model
- SQL: Strong expertise in writing and optimizing complex SQL queries.
- PySpark: Experience in using PySpark for data processing and transformation.
- ETL/ELT: Strong understanding of ETL/ELT processes and tools.
- Data Modeling: Knowledge of data modeling techniques and best practices.
- Data Governance: Familiarity with data governance, data quality, and data security practices.
Good-to-Have Skills:
- Experience with Azure DevOps for CI/CD pipelines.
- Knowledge in DLT (Delta Live Table ) in data bricks
Qualifications:
- Bachelor s or master s degree in computer science, Information Technology, or a related field.
- 5 to 9 Years of experience in data engineering, with a focus on Azure and Databricks.
- Relevant certifications such as Microsoft Certified: Azure Data Engineer Associate or Databricks Certified Associate Developer are
Disclaimer : This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.
Azure Databricks + Pyspark | | Engineering
Deloitte Consulting
naukri
Bengaluru
3-12 years Not Disclosed
Full time
01 May 2026
Top Skills:
Data QualityInformation TechnologyData ExtractionData ManagementAnalyticsData ModelingMicrosoftPerformance ManagementData ProcessingBusiness IntelligenceArchitectureAzureAzure Data FactoryAzure DevopsBatch ProcessingCi/cdData ArchitectureData ExtractionData GovernanceData IntegrationData LakeData ModelingData PipelineData ProcessingData QualityData RetrievalData SecurityDatabricksDevopsEtlFutureIndexingPysparkSparkSqlUnstructured Data
96
Get Personalized Job Matches with 1 Click
Job Description
Download Resume Job Title: Azure Data Engineer
Experience: 3 to 12 Years
Location: Bangalore, PAN India
Employment Type: Full-Time, Hybrid
Key Responsibilities:
1) Design and Development:
- Design, develop, and deploy scalable data pipelines using Databricks (PySpark, Spark SQL) , Azure Data Factory , and other Azure data services.
- Implement ETL/ELT processes to ingest, transform, and load data from various sources into data lakes and data warehouses.
- Optimize and tune data pipelines for performance and scalability.
2) Data Processing:
- Write and optimize complex SQL queries for data extraction, transformation, and analysis.
- Use PySpark for large-scale data processing and analytics.
- Implement data partitioning, bucketing, and indexing strategies for efficient data retrieval.
3) Data Integration:
- Integrate data from multiple sources, including structured, semi-structured, and unstructured data.
- Work with APIs, streaming data, and batch processing to ensure seamless data integration.
4) Data Governance and Quality:
- Implement data governance practices to ensure data quality, consistency, and security.
- Monitor and troubleshoot data pipelines to ensure data accuracy and availability.
5) Collaboration:
- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
- Work closely with DevOps teams to deploy and monitor data pipelines in production environments.
6) Documentation:
- Document data pipelines, workflows, and processes for knowledge sharing and future reference.
- Maintain up-to-date documentation on data architecture and data models.
Must-Have Skills:
- Databricks: Handson experience with Databricks for data processing, analytics, and Serverless SQLWH, Unity Catalog, Lakehouse, Medallion Architecture
- Azure Data Services: Proficiency in Azure Data Factory, Azure Data Lake Storage, and Azure SQL Database, Key Vault, Azure Pricing Model
- SQL: Strong expertise in writing and optimizing complex SQL queries.
- PySpark: Experience in using PySpark for data processing and transformation.
- ETL/ELT: Strong understanding of ETL/ELT processes and tools.
- Data Modeling: Knowledge of data modeling techniques and best practices.
- Data Governance: Familiarity with data governance, data quality, and data security practices.
Good-to-Have Skills:
- Experience with Azure DevOps for CI/CD pipelines.
- Knowledge in DLT (Delta Live Table ) in data bricks
Qualifications:
- Bachelor s or master s degree in computer science, Information Technology, or a related field.
- 5 to 9 Years of experience in data engineering, with a focus on Azure and Databricks.
- Relevant certifications such as Microsoft Certified: Azure Data Engineer Associate or Databricks Certified Associate Developer are
Disclaimer : This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.