Databricks Developer (Unity Catalog Migration) at DysrupIT in Remote

JOB SUMMARY

We are seeking a highly skilled Databricks Engineer with deep experience in Unity Catalog and enterprise-scale data platforms to support a major Oil & Gas Hydrocarbon Reporting initiative. The engineer will augment an existing delivery team and contribute across data engineering, governance, optimization, and analytics initiatives on the Databricks Lakehouse Platform.

The ideal candidate has hands-on experience implementing secure, governed data architectures using Unity Catalog and has worked on large-scale regulatory, production, or hydrocarbon reporting programs within the energy sector.

JOB RESPONSIBILITIES

• Design, develop, and maintain scalable data pipelines on the Databricks Lakehouse Platform.
• Implement and administer Unity Catalog for enterprise data governance, access control, lineage, and data sharing.
• Support hydrocarbon reporting workflows, including ingestion, transformation, validation, and reconciliation of production and operational data.
• Develop ETL/ELT pipelines using Spark, Delta Lake, and Delta Live Tables (Lakeflow experience is a plus).
• Optimize Databricks workloads for performance, reliability, and cost efficiency.
• Collaborate with business stakeholders to translate reporting requirements into scalable technical solutions.
• Implement data quality checks, monitoring, and automated validation frameworks.
• Work closely with solution architects and data engineers on platform enhancements and production support.
• Document technical designs, governance standards, and operational procedures.
• Participate in code reviews, CI/CD implementation, and DevOps best practices.

QUALIFICATIONS

• Bachelor's or Master's degree in Computer Science, Engineering, Information Systems, or a related discipline.
• 5+ years of data engineering experience with 3+ years of hands-on Databricks experience.
• Strong experience implementing and managing Unity Catalog in enterprise environments.
• Experience supporting Oil & Gas, Energy, or Hydrocarbon Reporting projects.
• Expertise with:
  • Databricks Lakehouse
  • Apache Spark (PySpark)
  • Delta Lake
  • SQL
  • Python
• Experience building enterprise ETL/ELT pipelines.
• Experience with Azure Databricks (AWS Databricks also considered).
• Strong understanding of data governance, security, metadata management, and lineage

PREFERRED QUALIFICATIONS:

• Experience with Delta Live Tables (DLT) or Lakeflow Pipelines.
• Knowledge of Unity Catalog features including ABAC, row/column-level security, external locations, catalogs, schemas, and lineage.
• Experience integrating with Azure Data Factory, Event Hubs, Kafka, or similar ingestion frameworks.
• Familiarity with MLflow, Databricks Workflows, and Databricks Asset Bundles.
• Experience working with hydrocarbon production, drilling, refinery, pipeline, or regulatory reporting datasets.
• Understanding of DevOps, Git, CI/CD, and Infrastructure as Code (Terraform preferred).

DESIRED SKILLS:

• Databricks Lakehouse Platform
• Unity Catalog Administration
• Delta Lake & Delta Live Tables
• Apache Spark / PySpark
• SQL & Python
• Data Governance & Metadata Management
• Performance Tuning & Cost Optimization
• Azure Cloud Services
• CI/CD & Git
• Agile Delivery