Data Engineer at Talent Muscle Pvt Ltd in Pune, Maharashtra

Data Engineer

• Work Location: Pune
• Experience: 3-5 years
• Immediate joiners only. Please mention LWD in the application, if serving.

Position Overview:

We are looking for a skilled Data Engineer to design, build, and maintain scalable data platforms and pipelines using Databricks, Microsoft Fabric and Azure data factory (ADF). The ideal candidate will have strong experience in modern data architectures, cloud infrastructure, and distributed data processing.

You will play a key role in building robust, high-performance data solutions that support analytics, reporting, and AI/ML use cases. This role requires close collaboration with data scientists, analysts, and platform teams to ensure reliable and efficient data flow across systems.

Key Responsibilities

· Design, develop, and maintain scalable data pipelines using Databricks (Spark), Azure Data Factory and Microsoft Fabric (Data Factory, Lakehouse, Warehouse)

· Build and manage end-to-end data solutions including ingestion, transformation, and serving layers

· Develop and optimize batch and streaming data pipelines

· Implement data storage solutions using Azure Data Lake (ADLS Gen2), Fabric Lakehouse, and Warehouse architectures

· Apply data modelling techniques (star schema, medallion architecture – bronze/silver/gold)

· Optimize data pipelines for performance, cost, and scalability

· Ensure data quality, governance, lineage, and monitoring across pipelines

· Develop reusable dataflows, notebooks, and ETL frameworks

· Collaborate with cross-functional teams to translate business requirements into technical solutions

· Support CI/CD and DevOps practices for data platform deployment

· Troubleshoot and resolve production data issues

Mandatory Skills:

· Strong hands-on experience with:

o Azure Databricks (Apache Spark, PySpark and PySQL)

o Azure Data Factory (ADF) and/or Microsoft Fabric

· Experience building ETL pipelines in Microsoft Fabric (Lakehouse, Dataflows Gen2, Pipelines), ADF and Databricks, etc

· Strong proficiency in Python and PySpark (PySQL preferred)

· Solid understanding of ETL/ELT concepts and data pipeline orchestration

· Experience with cloud data platforms (Azure preferred, AWS)

· Good knowledge of:

o Data Lake, Lakehouse and Warehouse architectures

o Delta Lake / Parquet formats

o Data modeling (dimensional & normalized)

· Experience with SQL and large-scale data processing

· Understanding of data platform architecture and infrastructure concepts:

o Networking basics (VNet, private endpoints)

o Security (IAM, RBAC, credentials, managed identities)

o Cost and performance optimization

· Familiarity with data quality frameworks and validation techniques

· Experience with code versioning (Git) and collaborative development

· Strong problem-solving and communication skills

Preferred Skills:

· Experience with Databricks and Microsoft Fabric end-to-end platform (Lakehouse, Warehouse, Real-Time Analytics)

· Understanding of Medallion Architecture (Bronze/Silver/Gold layers)

· Experience with MLOps / ML pipeline integration (MLflow, Azure ML, Databricks ML)

· Experience in streaming technologies (Structured Streaming, Event Hub, Kafka)

· Exposure to CI/CD tools (Azure DevOps, GitHub Actions)

· Knowledge of Infrastructure-as-Code (Terraform, ARM templates, Bicep)

· Familiarity with monitoring & observability tools (Azure Monitor, Log Analytics)

· Experience with data governance tools (Catalogue and Purview or similar)

· Exposure to Power BI integration with Fabric / Lakehouse

· Basic knowledge of containerization (Docker) and orchestration (Kubernetes) (nice to have)

Pay: Up to ₹1,200,000.00 per year

Application Question(s):

• What is your current CTC?
• What is your expected CTC?
• What is your notice period? Are you an immediate joiner?

Experience:

• data engineer: 3 years (Required)

Location:

• Pune, Maharashtra (Preferred)

Work Location: In person