Associate Data Engineer

Summary: Candid is a nonprofit that provides comprehensive data and insights about the social sector. They are seeking an Associate Data Engineer to support the day-to-day operations of their cloud data platform, focusing on pipeline maintenance, storage management, and platform observability. Responsibilities: - Pipeline Maintenance, Documentation, & Validation: Serve as the primary owner of ingestion pipelines and transformation table adjustments. Ensure continued, reliable data delivery and apply routine changes as business and schema needs evolve. Validate transformation outputs against expected results after schema or structural changes, documenting findings and escalating anomalies to the appropriate teams - Storage & Platform Support: Assist with scheduling compaction and cleanup jobs to maintain Iceberg table health and query performance. Support partition evolution and snapshot retention management to control storage growth - Observability & Metadata: Assist in implementing and maintaining CloudWatch metrics, alarms, and dashboards to ensure pipeline visibility. Contribute to tracking and reporting on platform performance metrics. Help maintain AWS Glue metadata refresh and statistics jobs that support query planning and optimization within the data platform - Schema Coordination: Assist with coordinating schema changes across ingestion and transformation layers to maintain consistency end to end. Collaborate with the Data Operations Engineer to communicate impacts and sequence changes safely - Infrastructure & Security: Support and maintain RBAC and ABAC (least privilege, standardized roles, and consistent tagging). Participate in access reviews and audits, documenting changes and escalating risks as needed Required Qualifications: - 1– 3 years of experience in data engineering, analytics engineering, or a closely related technical role; internships and relevant academic project work considered - Solid SQL skills, including writing, reading, and debugging queries against relational or columnar data stores - Familiarity with cloud data concepts: object storage (Amazon S3), columnar file formats (i.e. Parquet), data-interchange formats (JSON, XML), or open table formats (i.e. Apache Iceberg) - Experience with or exposure to distributed SQL query engines such as Trino or Starburst - Familiarity with AWS services such as S3 and Glue - Exposure to Apache Airflow, SSIS or another workflow orchestration platform - Experience writing or maintaining data pipelines in Python - Familiarity with on-prem relational data systems (i.e. Microsoft SQL Server) - Strong attention to detail, especially around data validation and output accuracy - Strong analytical and problem-solving skills - Excellent written and verbal communication skills; ability to document findings clearly for both technical and non-technical audiences - Ability to work independently and collaboratively as part of a distributed team - Willingness to perform other duties and special projects as needed/requested - Sensitivity and respect for racial, gender, sexual orientation, and cultural differences - Commitment to Candid's values: driven, direct, accessible, curious, and inclusive Required Skills: SQL, Amazon S3, Parquet, JSON, XML, Apache Iceberg, Trino, Starburst, AWS Glue, Apache Airflow, SSIS, Python, Microsoft SQL Server Benefits: Health insurance (medical, dental, vision), Retirement contribution with additional option for a match, Paid life insurance and AD&D, Paid leave time (PTO, compassionate leave, volunteer, holiday, parental), Short-term and long-term disability, Pre-tax transit, Flexible spending accounts, Supplemental insurance, Summer hours, Public Service Loan Forgiveness (PSLF) program eligible employer