Azure Databricks Data Engineer at Bhuvitech… · Hybrid -…

Role & Responsibilities;

We are looking for an experienced Azure Databricks Data Engineer with 6-8 years of experience in designing, developing, and supporting scalable data engineering solutions.

The candidate will be responsible for:

• Designing and developing scalable data pipelines using Azure Databricks, PySpark, and Azure Data Factory.
• Building ADF pipelines for ingestion, orchestration, scheduling, dependency management, and monitoring.
• Developing complex PySpark transformations for batch and large-volume data processing.
• Implementing Delta Lake and Lakehouse architecture using Bronze, Silver, and Gold layers.
• Designing ETL and ELT workflows for structured and semi-structured data.
• Writing and optimizing complex SQL queries.
• Improving Spark and PySpark performance through partitioning, caching, join optimization, cluster tuning, and efficient file handling.
• Implementing Delta Lake features such as schema evolution, MERGE, time travel, OPTIMIZE, VACUUM, and change data capture.
• Managing data access, governance, lineage, and security using Unity Catalog.
• Developing reusable frameworks for ingestion, transformation, auditing, logging, error handling, and data-quality validation.
• Implementing CI/CD processes for Databricks using Azure DevOps and Git.
• Troubleshooting production issues, pipeline failures, performance bottlenecks, and data-quality problems.
• Supporting deployment, monitoring, incident resolution, and root-cause analysis.
• Collaborating with architects, analysts, business stakeholders, and cross-functional teams.
• Preparing technical documentation, design documents, runbooks, and deployment guides.

Preferred Candidate Profile

• 68 years of overall experience in data engineering.
• Strong hands-on experience in Azure Databricks and PySpark development.
• Strong experience in Azure Data Factory pipeline development and orchestration.
• Good understanding of Databricks architecture, Spark architecture, clusters, jobs, notebooks, and workflows.
• Hands-on experience with Delta Lake, Lakehouse architecture, and Medallion architecture.
• Practical experience with Unity Catalog, data governance, role-based access control, security, and data lineage.
• Strong SQL development and query-optimization skills.
• Experience in Spark and PySpark performance tuning.
• Experience designing data models and end-to-end ETL/ELT frameworks.
• Hands-on experience implementing CI/CD using Azure DevOps.
• Strong problem-solving, debugging, and production-support skills.
• Good communication and stakeholder-management abilities.
• Candidates must be available to join immediately.
• Candidates must be genuinely interested and fully committed to the opportunity.
• Candidates should be comfortable working from any approved BSL location.
• Candidates should have a stable employment history and a low risk of offer withdrawal.