Senior QA Engineer - AI Products (6 months… at Arabic.AI · Doha

Senior QA Engineer - AI Products (6 months contract)

We are looking for a Senior QA Engineer to own quality across our AI products. You will test LLM-powered features end to end, build automated test and evaluation suites, and help ensure every release is accurate, safe, and reliable.

Responsibilities

• Define test strategies and plans for AI features across web, API, and backend.

• Evaluate model outputs for accuracy, consistency, hallucinations, bias, and safety.

• Build evaluation datasets, including edge cases and adversarial / red-team prompts.

• Develop and maintain automated UI, API, and regression tests integrated into CI/CD.

• Run regression testing on model, prompt, and configuration changes.

• Test performance, latency, error handling, and fallback behavior.

• Report defects clearly and provide release-readiness sign-off.

• Test non-deterministic outputs using scoring rubrics, LLM-as-judge, human evaluation, and defined pass/fail thresholds.

• Perform AI security testing, including prompt injection, jailbreaks, and PII / data leakage across chatbots, RAG, and agents.

• Work with ML engineers and product managers to define quality metrics and acceptance criteria.

• Where applicable, monitor production quality and model drift after release.

Requirements

7+ years of experience in QA / software testing, with strong test automation background.

• Proven experience testing AI/ML or LLM-based products (chatbots, RAG, agents, etc.).

• Hands-on scripting experience in Python and/or TypeScript/JavaScript.

• Hands-on with at least one UI automation framework (Playwright, Cypress, Selenium) and a test framework such as Pytest.

• API testing with Postman, REST Assured, or similar.

• Experience with LLM evaluation tools.

• CI/CD and version control: GitHub Actions, GitLab CI or Jenkins; Git.

• Test management and tracking: Jira, TestRail, or Zephyr.

• Excellent written English and clear bug reporting.

Nice to Have

• Performance/load testing (k6, JMeter, Locust).

• Understanding of prompt engineering, embeddings, and vector databases.

• Experience with AI safety, red-teaming, or content policy testing.

• Cloud experience (AWS, GCP, or Azure) and Docker.