Data Engineer at Glencore in New York, NY

Data Engineer – Commercial Engineering (New York, NY)

We are seeking a highly skilled and motivated Data Engineer to join our business; you will be part of a global Data Engineering team working on data engineering activities for our trading desks with a focus on the desks in New York.

The role reports to the Head of Data Architecture and Engineering.

Key Responsibilities:

• Design, implement, and maintain scalable data engineering pipelines using Python, Spark, and SQL, applying clean coding principles, and optimizing for performance in Azure and Databricks.
• Use good software engineering principles to enable modularity, reliability, and reusability of code.
• Orchestrate complex data workflows using frameworks like Airflow, Databricks workflows (or) equivalent, adopting a code-first approach.
• Provide data engineering support for trading analysts in NY.
• Use CI/CD tools like GitHub actions or equivalent for deploying data pipeline to higher environments efficiently.
• Implement data quality checks, validation frameworks, and monitoring systems to ensure data integrity and reliability.
• Design effective data models, optimize storage and retrieval processes, and efficiently handle semi-structured and unstructured data.
• Create and manage APIs for data access and integration and develop web scraping solutions using leading frameworks when necessary.
• Build scalable data ingestion pipelines for various external data feeds with different formats, velocity, and patterns.
• Collaborate closely with trading analysts in an agile environment, iterating data products in an efficient manner demonstrating strong understanding of the domain context.
• Apply strong analytical and logical problem-solving skills to address complex data engineering challenges and develop innovative solutions.
• Ensure data security and compliance with relevant regulations, implementing best practices for data protection and access control.
• Continuously optimize pipeline performance, evaluate new technologies, and drive innovation in the data engineering ecosystem.
• Maintain comprehensive documentation and contribute to the team's knowledge base, fostering a culture of continuous learning and improvement.
• Collaborate closely with data engineers in London to implement as per standardized guidelines and guardrails.

Qualifications and Requirements:

• Degree educated in either Computer Science, Mathematics, Engineering, or a related field.
• Strong hands-on experience with Python, Spark, SQL and strong knowledge of clean coding principles.
• Strong understanding of Data Governance best practices (DQ handling, Lineage..etc)
• Experience in handling various data ingestion patterns.
• Strong accountability and Entrepreneurialism in day-to-day work.
• Storing working experience with Data Orchestration (Airflow), Containerization (Docker, K8) and CI/CD tools.
• Desirable to have working knowledge in Microsoft Azure and Databricks.
• Advantageous to have experience working with Commodity/Energy trading data.
• Demonstrated ability to work effectively both independently and as part of a collaborative global team following internal Data engineering standards.
• Excellent problem-solving skills and a strong attention to detail.
• Strong stakeholder communication skills.

What We Offer:

• The expected base salary ranges from $120K -$180K. Salary offers are based on a wide range of factors including relevant skills, training, experience, education, and, where applicable, certifications and licenses obtained. Market and organizational factors are also considered. In addition to salary and a generous employee benefits package, successful candidates are eligible to receive a discretionary bonus.