Databricks Engineer at CGI in Hybrid - Hyderabad

Role & responsibilities

Position Description: Design, develop, and maintain scalable ETL/ELT pipelines using Databricks and PySpark.
• Develop data processing solutions using Python and Spark for large-scale data transformation.
• Write complex SQL queries for data extraction, transformation, validation, and reporting.
• Build and optimize data workflows to improve performance, scalability, and reliability.
• Work with structured and semi-structured data from multiple data sources.
• Collaborate with data analysts, business stakeholders, and development teams to understand data requirements.
• Perform data quality checks, troubleshooting, and root cause analysis for data issues.
• Optimize Spark jobs and SQL queries for improved performance.
• Follow coding standards, documentation, and best practices for data engineering.
• Participate in code reviews and support production deployments.

Responsibilities:
• Minimum 3 years of experience in Data Engineering.
• Strong hands-on experience with Databricks.
• Good experience in PySpark and Apache Spark.
• Strong programming skills in Python.
• Proficiency in SQL, including complex queries, joins, window functions, and performance tuning.
• Experience in developing ETL/ELT pipelines.
• Knowledge of data warehousing concepts and data modeling.
• Experience with Git or other version control systems.
• Strong analytical, problem-solving, and debugging skills.

Must-Have Skills:
• Strong experience Databricks
• Hands-on experience Data & Delta Lake
• Experience with PySpark and Apache Spark.
• Experience with Python.
• Experience with ETL/ELT pipelines.
• Excellent communication and leadership skills

Good-to-Have Skills:
• data warehousing concepts
• Experience with Git or other version control systems.
• Background in software development or system architecture