Databricks Data Engineer at Diaconia in Silver Spring, MD

Description

We are seeking a Databricks Data Engineer with strong hands-on experience in Python and PySpark to support a fast-paced, high-impact delivery environment. This role requires someone who can quickly step in, take ownership, and deliver scalable data solutions with minimal ramp-up. 

Key Responsibilities 

· Design, develop, and optimize data pipelines and ETL workflows using Databricks

· Build scalable data processing solutions leveraging PySpark and Python 

· Work with large-scale datasets across cloud platforms (AWS and/or Azure) 

· Implement and maintain Medallion Architecture (Bronze, Silver, Gold layers) 

· Ensure data quality, performance optimization, and reliability of data pipelines 

· Collaborate with cross-functional teams to support data-driven initiatives 

Required Qualifications  

· 3 to 5 years of experience in Databricks is required. 

· Strong hands-on experience with Databricks, Python, PySpark, and SQL 

· Proven experience building and optimizing ETL pipelines 

· Experience working with large-scale data environments 

· Solid understanding of cloud platforms (AWS and/or Azure) 

· Familiarity with Medallion architecture (Delta Lake) 

· Ability to work independently in fast-paced, deadline-driven environments 

What We’re Looking For 

· Someone who can hit the ground running with minimal onboarding 

· Strong sense of ownership and accountability 

· Ability to deliver under tight timelines without compromising quality 

· Proactive problem-solver with a delivery-first mindset Nice to Have 

· Experience with Delta Lake

· Exposure to data governance and data quality frameworks 

· Prior experience in federal or regulated environments

Preferred Additional Qualifications:  

· Experience with Delta Lake 

· Exposure to data governance and data quality frameworks 

· Prior experience in federal or regulated environments

Requirements

Required Qualifications: :

Education: 

• Bachelor's degree in information systems, computer science, or other related field. Will accept a suitable combination of education, training, or experience

Clearance: 

• Clearance: Federal Health Agency within the U.S. Department of Health and Human Services Public Trust clearance 

Onsite Requirement: 

• Location: Hybrid - locals to DMV area 

Professional Experience 

• 3 to 5 years of experience in Databricks is required. 
• Strong hands-on experience with Databricks, Python, PySpark, and SQL 
• Proven experience building and optimizing ETL pipelines 
• Experience working with large-scale data environments 
• Solid understanding of cloud platforms (AWS and/or Azure) 
• Familiarity with Medallion architecture (Delta Lake) 
• Ability to work independently in fast-paced, deadline-driven environments