AI Eval Engineer at Hiringeye Solutions in Bengaluru

We are looking for an AI Eval Engineer to build evaluation systems and tools for LLM-powered education products. The role involves Python development, AI workflow testing, model evaluation, data analysis, and improving the quality, accuracy, and safety of AI outputs.

Key Responsibilities

• Build and maintain LLM evaluation pipelines and testing tools.
• Develop Python scripts for batch testing, model comparison, regression testing, and data analysis.
• Create test datasets, golden datasets, rubrics, and evaluation criteria.
• Evaluate AI workflows involving prompts, RAG, structured outputs, tool calling, and multi-step reasoning.
• Identify hallucinations, quality issues, regressions, and recurring AI failure patterns.
• Collaborate with Engineering, Product, QA, and Curriculum teams to improve AI products.

Required Skills

• 3+ years of experience in Software Development, AI/ML, AI Evaluation, QA Automation, or Data Analysis.
• Strong Python and scripting skills.
• Experience with LLM APIs, JSON, datasets, notebooks, and API-based workflows.
• Understanding of AI evaluation, prompt testing, regression testing, and quality measurement.
• Strong analytical and debugging skills.

Preferred Skills

• Experience with LLM-as-a-Judge, RAG, tool calling, and multi-step AI workflows.
• Experience in EdTech, curriculum, tutoring, or educational products.
• Familiarity with dashboards, annotation tools, Git, and experiment tracking.