Senior Data Engineer (1043) - DataSF - Office of… at City and… · San…

Company Description

Department: DataSF

Job class: 1043 Senior Data Engineer

Role type: Permanent Exempt

Hours: Full-time. The department has a hybrid work schedule.

About:

Permanent Exempt: This is an Exempt position excluded by the Charter from the competitive Civil Service examination process, pursuant to the City and County of San Francisco, Charter Section 10.104. It is considered "at will" and shall serve at the discretion of the Appointing Officer. 

Specific information regarding this recruitment process is listed below:

• Application Opening: Monday, June 1, 2026
• Application Closing: Interested candidates are encouraged to apply as soon as possible, as this job announcement will close at any time, but not earlier than Sunday, June 21, 2026 at 11:59pm PST

As part of the application process, you must also complete a Supplemental Questionnaire using this link by the filing deadline: https://forms.cloud.microsoft/g/0nKeS3Mdry

The Office of the City Administrator and its 25+ divisions and departments operate core internal and public-facing services in San Francisco.

The Office of the City Administrator’s Mission and Vision

Our vision is to lead the nation in public administration and to enable City departments to effectively deliver critical public services. We aim to help the city run better, to connect San Francisco residents and constituents to the vital public services they seek, and to create a meaningful and diverse work culture that is the place of choice for people who are invested in a career in public service.

We are committed to ensuring that City services are inclusive, efficient, equitable, and culturally competent for San Franciscans of all races, ethnic backgrounds, religions, and sextual orientations. This commitment requires comprehensive review and thorough analysis of existing practices and policies to remove barriers to real inclusion. We are also committed to ensuring that we have a safe, equitable, and inclusive workplace for individuals of all races. This includes creating opportunities for hiring, promotion, training, and development, for all employees, including but not limited to, Black, Indigenous, and people of color (BIPOC).

To learn more about our departments, divisions, and programs, visit: https://sf.gov/departments/city-administrator

About DataSF:

Want to build robust and scalable data infrastructure to power data-driven City services? Join the DataSFteam to empower and expand the use of data in government!

Company Description

DataSF’s mission is to transform the way San Francisco works with data. We believe that data, when harnessed effectively, can improve transparency, resident engagement, and government performance. We are committed to removing barriers and making it easier for all people to access services and knowledge, and we actively seek team members whose diverse life experiences reflect the residents we serve.

Job Description

DataSF is seeking a Senior Data Engineer with 3+ years of experience to join our growing team. Reporting to the Principal Data Engineer, you will be instrumental in designing, building, and maintaining the City's data infrastructure, enabling robust data pipelines and reliable data access for analytical and operational needs. This is an exciting position for someone eager to apply advanced data engineering techniques to complex urban challenges, contributing directly to San Francisco's commitment to efficient, equitable, and ethical service delivery.

Learn more about DataSF’s recent work on our blog. If you are an entrepreneurial and passionate data enthusiast, join our team to improve government through good use of data!

Essential duties include, but are not limited to, the following:

• Platform Administration: Manage our central Snowflake data warehouse, including access control, security policies, resource monitoring, performance tuning, and cost optimization. Administer our platform with a focus on data democratization and accessibility, while protecting privacy and security.
• Pipeline Development: Build and maintain scalable and resilient pipelines to ingest and structure data from diverse sources. Design infrastructure to support both streaming and batch processes, and both structured and unstructured data sources.
• Infrastructure as Code (IaC): Use Terraform to define, deploy, and manage data infrastructure, ensuring our pipelines are reproducible, version-controlled, and production-ready.
• Best Practices & Innovation: Champion and implement best practices for documentation, data modeling, warehouse architecture, SQL optimization, and testing. Think creatively to find new ways to improve our data platform's capabilities and efficiency. Provide guidance to department partners on data engineering best practices.
• Collaboration: Work closely with data scientists, analysts, product managers, software engineers, and nontechnical stakeholders in diverse domains to understand data requirements and build solutions that meet their needs.
• Monitoring & Support: Proactively monitor the health of the data platform and pipelines, troubleshoot issues, and ensure high standards of data quality and availability.

Qualifications

Minimum Qualifications:

Education:

An associate degree in computer science, computer engineering, information systems, or a closely related field from an accredited college or university OR its equivalent in terms of total course credits/units [i.e., at least sixty (60) semester or ninety (90) quarter credits/units with a minimum of twenty (20) semester or thirty (30) quarter credits/units in one of the fields above or a closely-related field].

Experience:

Three (3) years of experience analyzing, installing, configuring, enhancing, and/or maintaining the components of an enterprise network.

Substitution:

Additional experience as described above may be substituted for the required degree on a year-for-year basis (up to a maximum of two (2) years). One (1) year is equivalent to thirty (30) semester units/ forty-five (45) quarter units with a minimum of 10 semester / 15 quarter units in one of the fields above or a closely related field.

Completion of the 1010 Information Systems Trainee Program may be substituted for the required degree.

Verification:Please make sure it is clear in your application exactly how you meet the minimum qualifications. Applicants will be required to submit verification of qualifying education and experience during the recruitment and selection process. For information on how to verify experience and/or education requirements, including verifying foreign education credits or degree equivalency, please visit Verification of Experience and/or Education.

Note:

• Falsifying one’s education, training, or work experience or attempted deception on the application may result in disqualification for this and future job opportunities with the City and County of San Francisco.
• One-year full-time employment is considered equivalent to 2000 hours (2000 hours of qualifying work experience is based on a 40 hour work week). Any overtime hours that you work above forty (40) hours per week are not included in the calculation to determine full-time employment
• Applicants must meet minimum qualification requirements by the final filing date unless otherwise noted.

Desirable Qualifications

• Technical Knowledge
  • Hands-on experience administering and developing on managed cloud data platforms such as Snowflake, BigQuery, or Databricks.
  • Demonstrated expertise in writing advanced, performant SQL, and using tools like dbt for SQL-based data transformation and modeling.
  • Strong programming skills in Python (with libraries like pandas, PySpark) for data processing and automation.
  • Proficiency with an Infrastructure as Code tool, with a preference for Terraform.
  • Experience building and deploying data pipelines using orchestration tools like Azure Data Factory, Airflow, Dagster, or similar technologies.
  • Deep understanding of data warehousing concepts, data modeling, and modern ELT principles.
  • Understanding of data governance, data security, and data privacy principles.
  • Experience with real-time data streaming technologies (e.g., Kafka, Kinesis, Snowpipe).
  • Experience deploying and managing data pipelines for machine learning models.
• Collaboration and Communication Skills
  • Strong problem-solving skills with ability to design practical and effective data solutions
  • Excellent verbal and written communication skills, including the ability to explain technical concepts to non-technical stakeholders
  • A collaborative mindset with enthusiasm to work across diverse, cross-functional teams
• Mission Alignment
  • Commitment to equity, transparency, and ethical data use
  • Passion for public service and using data to improve government services
  • Empathy for San Francisco’s diverse communities and a drive to make data and services and more accessible for SF residents
  • Interest or experience in public sector data or social impact work

Additional Information

Selection Procedures

The selection process will include evaluation of applications in relation to minimum requirements and assessment of candidates’ job-related knowledge, skills and abilities. Depending on the number of applicants, the Department may establish and implement additional screening mechanisms to evaluate candidate qualifications. This typically includes an oral interview and/or a written or performance exercise.

If this becomes necessary, only those applicants whose qualifications most closely meet the Department needs will be invited to continue in the selection process. Applicants meeting the minimum requirements are not guaranteed advancement in the selection process.

To find Departments which use this classification, please see: https://sfdhr.org/sites/default/files/documents/Forms-Documents/Position-Counts-by-Job-Codes-and-Department-FY-2022-23.pdf.

Additional Information Regarding Employment with the City and County of San Francisco:

• Information About The Hiring Process
• Conviction History
• Employee Benefits Overview
• Equal Employment Opportunity
• Disaster Service Worker
• ADA Accommodation
• Veterans Preference
• Right to Work
• Copies of Application Documents
• Diversity Statement

How to Apply:

Applications for City and County of San Francisco jobs are only accepted through an online process. Visit https://careers.sf.gov and begin the application process.

• Interested candidates are encouraged to apply as soon as possible, as this job announcement will close at any time, but not before Sunday, June 21, 2026 at 11:59 PM, PST
• Select the "Apply Now" button and follow instructions on the screen
• As part of the application process, you must also complete a Supplemental Questionnaire using this link: https://forms.cloud.microsoft/g/0nKeS3Mdry

For best practices on the application process, please visit Apply for Jobs in the City and County of San Francisco Best Practices Guide. Applicants may be contacted by email about this announcement and, therefore, it is their responsibility to ensure that their registered email address is accurate and kept up-to-date. Also, applicants must ensure that email from CCSF is not blocked on their computer by a spam filter. To prevent blocking, applicants should set up their email to accept CCSF mail from the following addresses @sfgov.org, @sfdpw.org, @sfport.com, @flysfo.com, @sfwater.org, @sfdph.org, @asianart.org, @sfmta.com, @sfpl.org, @dcyf.org, @first5sf.org, @famsf.org, @ccsf.edu, @smartalerts.info, and @smartrecruiters.com).

Applicants will receive a confirmation email that their online application has been received in response to every announcement for which they file. Applicants should retain this confirmation email for their records. Failure to receive this email means that the online application was not submitted or received.

All your information will be kept confidential according to EEO guidelines.

HR Analyst Information: If you have any questions regarding this recruitment or application process, please contact Connie Poon at Connie.Poon@SFGov.org

Condition of Employment:

The City and County of San Francisco encourages women, minorities and persons with disabilities to apply. Applicants will be considered regardless of their sex, race, age, religion, color, national origin, ancestry, physical disability, mental disability, medical condition (associated with cancer, a history of cancer, or genetic characteristics), HIV/AIDS status, genetic information, marital status, sexual orientation, gender, gender identity, gender expression, military and veteran status, or other protected category under the law.

The City and County of San Francisco encourages women, minorities and persons with disabilities to apply. Applicants will be considered regardless of their sex, race, age, religion, color, national origin, ancestry, physical disability, mental disability, medical condition (associated with cancer, a history of cancer, or genetic characteristics), HIV/AIDS status, genetic information, marital status, sexual orientation, gender, gender identity, gender expression, military and veteran status, or other protected category under the law.

The City and County of San Francisco encourages women, minorities and persons with disabilities to apply. Applicants will be considered regardless of their sex, race, age, religion, color, national origin, ancestry, physical disability, mental disability, medical condition (associated with cancer, a history of cancer, or genetic characteristics), HIV/AIDS status, genetic information, marital status, sexual orientation, gender, gender identity, gender expression, military and veteran status, or other protected category under the law.