Director Data Engineer at Embarkgcc Services in Hyderabad

Summary:

We are seeking a highly skilled and motivated  Sr  Data Engineer to join our innovative team. As a Data Engineer, you will be responsible for designing, building, and maintaining scalable data pipelines and infrastructure to support our enterprise-wide data-driven initiatives. You will collaborate closely with cross-functional teams to ensure the availability, reliability, and performance of our data systems and solutions.

This role involves working across modern data engineering frameworks, cloud platforms, and distributed environments to enable efficient data integration, transformation, governance, and consumption across the organization. You will contribute to architectural decisions, lead complex data engineering tasks, and support best practices for high quality, secure, and scalable data solutions.

Responsibilities:

Data Pipeline Development

• Design, implement, and optimize end-to-end data pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data.
• Develop robust ETL (Extract, Transform, Load) processes to integrate data from diverse sources into our data ecosystem.
• Implement data validation and quality checks to ensure accuracy and consistency across systems.

Data Modeling and Architecture

• Design and maintain data models, schemas, and database structures for analytical and operational use cases.
• Optimize data storage and retrieval mechanisms to support performance and scalability.
• Evaluate and implement data storage solutions including relational databases, NoSQL systems, data lakes, and cloud storage platforms.

Data Integration and API Development

• Build and maintain integrations with internal and external data sources and APIs.
• Implement RESTful APIs and web services for secure and efficient data access and consumption.
• Ensure compatibility, standardization, and interoperability across systems.

Data Infrastructure Management

• Configure and manage data infrastructure components including databases, warehouses, lakehouses, and distributed computing frameworks.
• Monitor system performance, identify bottlenecks, troubleshoot issues, and implement optimizations.
• Implement data security controls, governance, and access management policies to protect sensitive information.

Collaboration and Documentation

• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver effective technical solutions.
• Document technical designs, workflows, data flow diagrams, and best practices to enable transparency and knowledge sharing.
• Provide technical guidance, mentorship, and support to team members and cross-functional partners.

Requirements:

• Bachelor’s degree in computer science, Engineering, Information Systems, or a related field.
• 10+ years of proven experience in data engineering, software development, or related technical roles.
• 10+ years of experience in programming languages commonly used in data engineering (Python, Java, SQL, Stored Procedures, Scala, etc.).
• 10+ years of experience with database systems, data modeling, and advanced SQL.
• 10+ years of experience with ETL tools such as SSIS, Snowflake, Databricks, Azure Data Factory, Stored Procedures, etc.
• Experience with big data technologies such as Hadoop, Spark, Kafka, etc.
• 7+ years of experience working with cloud platforms like Azure, AWS, or Google Cloud.
• Strong analytical, problem-solving, and debugging skills with high attention to detail.
• Excellent communication and collaboration skills in a team-oriented, fast-paced environment.
• Ability to adapt to rapidly evolving technologies and business requirements.