Data Architect/Engineer at Bespoke Technologies in Chantilly, VA

BT-271 – Data Architect/Engineer
Location: Chantilly, VA (fully on-site, no remote option)

 

• The Contractor shall conduct comprehensive assessments of existing data pipelines, infrastructure, and data flows including integrations with operational systems like ServiceNow, network management platforms, and business applications to identify technical debt, bottlenecks, and reliability issues.
• The Contractor shall evaluate current data architecture against industry best practices and organizational needs; develop technical recommendations and roadmaps for data infrastructure improvements.
• The Contractor shall design, build, and maintain production-grade data pipelines using orchestration tools such as Airflow or Prefect.
• The Contractor shall develop robust ETL(Extract-Transform-Load/ELT (Extract-Load Transform?) processes from diverse sources: SaaS platforms, network management systems, databases, APIs, files, and streams.
• The Contractor shall build API integrations handling authentication (OAuth, API keys, and Single
• Sign-On (SSO), rate limiting, pagination, retry logic, and error handling.
• The Contractor shall extract data from systems not designed for export; reverse-engineer undocumented data structures and relationships.
• The Contractor shall handle semi-structured data (JSON and XML); and transform into structured datasets with consistent schemas.
• The Contractor shall design dimensional models, data warehouses, and data marts following industry methodologies.
• The Contractor shall create conceptual, logical, and physical data models optimized for query performance and storage efficiency.
• The Contractor shall implement slowly changing dimensions and other data warehousing patterns.
• The Contractor shall establish naming conventions, data standards, and modeling best practices.
• The Contractor shall implement comprehensive data quality checks, validation rules, and automated monitoring with alerting.
• The Contractor shall build error handling, failure recovery, logging, and observability into all processes.
• The Contractor shall optimize pipelines for performance, cost, and resource utilization.
• The Contractor shall develop reusable components and frameworks; refactor legacy pipelines for reliability.
• The Contractor shall build and maintain data infrastructure on cloud platforms (Amazon Web Services
• (AWS), Microsoft Azure, or Google Cloud Platform (GCP) using infrastructure-as-code using Terraform and CloudFormation.
• The Contractor shall implement CI/CD pipelines, version control (Git), and automated testing frameworks.
• The Contractor shall manage database performance tuning, indexing, partitioning, and capacity planning.
• The Contractor shall establish backup, recovery, security controls, access controls, and compliance measures.
• The Contractor shall partner with analysts, software developers, and business stakeholders to translate requirements into technical solutions.
• The Contractor shall create comprehensive documentation for systems, processes, and integrations.
• The Contractor shall provide technical guidance on data availability and proper usage; enable self-service access.
• The Contractor shall troubleshoot pipeline failures, performance issues, and data discrepancies; perform root cause analysis.

• Demonstrated experience designing, building, and maintaining production data pipelines using orchestration tools such as Apache Airflow or similar.
• Demonstrated experience with SQL skills including complex queries, optimization, and performance tuning across multiple database platforms.
• Demonstrated experience integrating data from Sponsor SaaS platforms and operational systems via APIs, including handling authentication, pagination, and rate limiting.
• Demonstrated experience working with semi-structured data (JSON and XML) from API responses and transforming into structured datasets.
• Demonstrated experience with developing robust API integrations with proper error handling and retry logic.
• Demonstrated experience working with systems that have limited documentation or vendor-specific data models.
• Demonstrated experience with dimensional modeling and data warehouse design patterns.
• Demonstrated proficiency in Python for data engineering including working with data processing libraries.
• Demonstrated experience with cloud data platforms such as AWS, Azure, or GCP, including data services and infrastructure.
• Demonstrated experience implementing ETL/ELT processes from diverse data sources.
• Demonstrated experience with version control (Git) and software engineering best practices.
• Demonstrated experience with strong problem-solving and troubleshooting skills for complex data pipeline issues.
• Demonstrated experience implementing data quality checks and validation frameworks.
• Demonstrated experience translating business requirements into technical data solutions.
• Demonstrated experience in having a proven track record of delivering reliable, scalable data infrastructure.

• Demonstrated experience with ServiceNow APIs, data models, and integration patterns.
• Demonstrated experience with network management or IT operations systems data extraction.
• Demonstrated experience with Forward Networks, NetIM, SolarWinds, or similar network management platforms.
• Demonstrated experience and knowledge of ITSM (Information Technology Service Management), ITOM (Information Technology Operations Management), and CMDB (Configuration Management Database) data structures and relationships.
• Demonstrated experience with API gateway platforms and API management tools.
• Demonstrated experience with Apache Spark, particularly PySpark, for distributed data processing.
• Demonstrated experience with DBT (data build tool) for transformation workflows.
• Demonstrated experience with infrastructure-as-code tools such as Terraform or CloudFormation.
• Demonstrated experience implementing CI/CD (Continuous Integration/Continuous Delivery) pipelines for data engineering code.
• Demonstrated experience and knowledge of streaming data technologies such as Kafka, Kinesis, or similar platforms.
• Demonstrated experience with data quality platforms such as Great Expectations, Soda, or Monte Carlo.
• Demonstrated experience implementing data observability and monitoring solutions.
• Demonstrated experience and knowledge of Data Vault or other advanced modeling methodologies.
• Demonstrated experience with containerization (Docker) and orchestration (Kubernetes) for data workloads.
• Demonstrated experience with reverse ETL and operational analytics patterns.
• Demonstrated experience with data governance platforms and metadata management tools.
• Demonstrated experience with multiple cloud platforms and multi-cloud architectures.
• Demonstrated experience mentoring or leading data engineering initiatives.