Systems Engineer/Senior Data Engineer - Splunk,… at KDA… · Herndon

***Security Clearance: Must possess an active TS/SCI with Full Scope Polygraph***

KDA is seeking a highly skilled and experienced Systems Engineer/Senior Data Engineer with a strong background in designing, implementing, and optimizing data pipelines and solutions for critical security and operational platforms, specifically Splunk, ServiceNow, and AppDynamics. The ideal candidate will be a proactive, problem-solving individual with a proven track record of managing complex data ingestion, normalization, and correlation processes within highly visible, operationally driven environments. This role requires an individual who thrives in a collaborative setting, possesses excellent communication skills, and is dedicated to continuous improvement and innovation.

Key Responsibilities:

• Splunk Data Engineering:
  • Design, engineer, and maintain robust Splunk infrastructures, including clustered environments, for large-scale data ingestion, correlation, and reporting.
  • Automate complex data ingestion methods (e.g., S3, syslog, JSON, APIs) from diverse sources across multiple enclaves.
  • Develop and implement methods for data tagging and cataloging to ensure compliance with evolving security standards and facilitate efficient data discovery.
  • Optimize data ingest performance and efficiency across various network environments. Familiar with Technical Add-ons.
  • Parse and normalize non-standard data sets to enable comprehensive analysis and correlation within Splunk.
  • Develop and refine Splunk queries, dashboards, and reports to visualize security events, infrastructure health, and operational metrics.
  • Collaborate with IT operations and cyber security teams to enrich data sets, ascertain cyber threats, and bolster security posture.
  • Maintain ITSI and SIEM-like tools and custom content within virtualized environments.
  • Perform tuning and filtering of events and information, creating custom views and content.
  • Familiar with UBA and Splunk.
  • Collaborate with cross-functional teams to design and implement data integrations between various security and operational tools (including Splunk and AppDynamics) and ServiceNow.
  • Develop and maintain data pipelines to ensure accurate and timely flow of security incidents, alerts, and operational metrics into ServiceNow for incident management, problem management, and reporting.
  • Assist in defining and implementing data models within ServiceNow to support security operations and compliance initiatives.
  • Work with third-party services for design review and optimal deployment configuration for enterprise cloud service utilization (relevant to integrations).
  • Design and implement data collection strategies for AppDynamics, ensuring comprehensive monitoring of application performance and infrastructure.
  • Integrate AppDynamics data with Splunk for centralized visibility and correlation with other security and operational logs.
  • Troubleshoot problematic service deployments and data flows, utilizing forensic tools and audit log review (relevant to monitoring and analysis).
  • Develop methods to leverage AppDynamics data for identifying potential risks and optimizing application performance.
• General Data Engineering & Systems Expertise:
  • Collaborate with mission partners to develop long-term enterprise audit solutions and normalize non-standard data sets.
  • Engineer and maintain secure virtualized and cloud environments for data platforms.
  • Deploy and harden servers running Linux OS in accordance with CIS and other STIG guidelines.
  • Develop runbooks, SOPs, and documentation for new processes and systems.
  • Perform liaison duties between service providers and clients to bridge communication gaps and ensure adherence to SLAs.
  • Review and evaluate data integrity and develop use cases for various data sets.
  • Maintain system baselines and configuration management for data engineering tools.
  • Contribute to the development of plans to safeguard data against unauthorized modification, destruction, or disclosure.
  • Strong understanding of cyber security principles and experience with various security tools (e.g., Next-Gen Firewalls, IPS/IDS, Tenable Nessus, Rapid7 Nexpose, McAfee EPO, Symantec SEP).

Required Skills & Experience:

• 20+ years of progressive experience in Information Technology and Security, with a strong focus on data engineering and systems integration.
• Security Clearance: Candidate must possess an active TS/SCI with Full Scope Polygraph
• Demonstrated expertise in engineering and maintaining large-scale Splunk environments, including data ingestion, parsing, normalization, and content development.
• Experience with automating complex data ingestion methods (e.g., S3, syslog, JSON, APIs).
• Strong understanding of data tagging, cataloging, and data governance best practices.
• Proficiency with Linux OS administration and hardening.
• Familiarity with cloud security principles and deploying commercial services into protected/secured enclaves (e.g., AWS).
• Experience with SIEM solutions and their implementation, configuration, and maintenance.
• Strong scripting skills (e.g., BASH, Python, PowerShell).
• Excellent collaboration and communication skills, with the ability to work effectively in small teams and large collaborative efforts.
• Ability to troubleshoot complex technical issues and perform root cause analysis.
• Proven ability to develop and maintain documentation (runbooks, SOPs).

Desired Skills (Plus, but not required):

• Experience with ServiceNow platform administration, development, or integration.
• Experience with AppDynamics for application performance monitoring and data collection.
• Experience with configuration management tools - Git, Ansible