AI Test Engineer (SME) at NeatDevs LLC in Ashburn, VA

NeatDevs, LLC is a dynamic federal contracting company dedicated to providing cutting-edge technology solutions across custom software development, artificial intelligence, cybersecurity, and data analytics. We excel by delivering innovative, secure, and mission-critical applications tailored specifically to the unique needs of federal agencies.

At NeatDevs, our standout culture emphasizes collaboration, continuous learning, and a commitment to excellence. Our employees thrive in an environment that promotes innovation, supports professional growth, and fosters work-life balance. We proudly leverage agile methodologies and advanced technology stacks to drive impactful solutions that protect national interests and enhance government efficiency.

Join NeatDevs and become part of a passionate team making meaningful contributions to our nation’s security and technological advancement. Our employees benefit from:

An engaging, collaborative, and inclusive team environment

Robust professional development and mentorship opportunities

Excellent work-life balance and flexible work arrangements

Competitive compensation and comprehensive benefits, including immediate 401K matching

If you’re looking to grow your career with a company that values innovation, integrity, and teamwork, NeatDevs is the place for you!

What you'll do

• Define and own test strategy, standards, and quality gates for AI components
• Build evaluation approaches for non-deterministic output: golden datasets, scoring rubrics, acceptance thresholds, LLM-as-judge evaluation
• Test agents end to end: task completion, tool selection and trajectory, guardrails, human-in-the-loop checkpoints
• Test MCP servers and clients: schema conformance, error handling, authentication, least-privilege access
• Perform adversarial testing: prompt injection, jailbreaks, data leakage; measure latency, token usage, and cost
• Maintain automated suites in CI/CD; track test activity in Jira/Xray; mentor the team on AI testing

Required

• U.S. citizenship and ability to obtain a DHS/CBP Public Trust; must live in the DC/MD/VA area
• Bachelor's degree (BA/BS) in Computer Science or related field
• 10+ years of software QA and testing experience, including 5+ years in test automation
• 2+ years hands-on testing AI/ML or LLM-based applications (chat assistants, RAG, agents, tool integrations)
• Strong Python test development (pytest); strong API testing (REST, JSON schemas, auth)
• Working knowledge of agent concepts and the Model Context Protocol (MCP)
• CI/CD test integration (GitLab, Jenkins)

Nice to have

• Gemini / Vertex AI; Google ADK or LangGraph agent evaluation
• LLM eval frameworks (Vertex AI GenAI eval, promptfoo, DeepEval, Ragas)
• OWASP LLM Top 10, NIST AI RMF; Section 508 (ANDI); ISTQB AI Testing certification

Pay and benefits

$104,000 - $119,000 per year depending on experience and clearance status. Full-time W-2 role with health, dental, and vision insurance, life insurance, 401(k), paid time off, and paid federal holidays.

NeatDevs LLC is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or protected veteran status.