We are looking for a Senior AI Engineer to lead the development and optimisation of an LLM evaluation platform supporting a production AI-powered shopping assistant. The role focuses on building LLM evaluation pipelines, LLM-as-a-judge systems, golden datasets, and automated regression tests using Python, SQL, and LLM observability tools such as Langfuse, Braintrust, or Weave, while analysing real user conversations to identify hallucinations, quality issues, and potential risks. The engineer will take technical ownership of the evaluation platform, collaborate with product and engineering teams, and improve AI agents, prompt and context engineering, guardrails, and monitoring to ensure reliable, measurable, and continuously improving LLM performance.
Responsibilities
• Develop and maintain LLM evaluation pipelines
• Build LLM-as-a-judge solutions to assess AI conversations
• Create evaluation datasets and golden test sets
• Analyse real customer conversations and identify recurring problems
• Measure AI response quality and compare model performance
• Build automated regression tests for LLM applications
• Develop monitoring and observability for AI systems
• Identify hallucinations, incorrect responses and unusual user behaviour
• Work on AI guardrails, adversarial testing and abuse detection
• Improve prompts, context management and agent workflows
• Work closely with product managers and engineering teams
• Take technical ownership of the evaluation platform
Requirements
Must have
• Minimum 5 years of software engineering experience
• Minimum 2 years working with LLM systems in production
• Strong Python development
• Hands-on experience evaluating LLM applications
• LLM-as-a-judge and evaluation frameworks
• Prompt engineering and context engineering
• Experience with AI agents, tool calling and multi-turn conversations
• LLM observability tools such as Langfuse, Braintrust, Weave or similar
• SQL and experience working with large datasets
• CI/CD, testing, Docker or other container technologies
• Cloud infrastructure experience
• Experience using AI development tools such as Cursor, Claude Code or GitHub Copilot
• Good English and Polish
Nice to have
• Human evaluation and calibration of LLM results
• Golden datasets and regression testing for generative AI
• Agent orchestration and automated workflows
• MCP and agent-to-agent integrations
• AI red-teaming, jailbreak testing and guardrails
• Experience improving LLM applications using production feedback
• E-commerce, search or recommendation systems
• Experience as a Technical Lead or Staff Engineer
Benefits
• Flexible hybrid mode (1–2 days per week in the office in Warsaw/Kraków)
• Solid, competitive salary
• Work in a multinational environment on international projects
• Comprehensive healthcare
• Long-term B2B contract with a stable project pipeline
