LLM Quality Engineer at AECOM in Oslo, Oslo, Norway

companyDescription Work with Us. Change the World. At AECOM, we're delivering a better world. Whether improving your commute, keeping the lights on, providing access to clean water, or transforming skylines, our work helps people and communities thrive. We are the world's trusted infrastructure consulting firm, partnering with clients to solve the world’s most complex challenges and build legacies for future generations. There has never been a better time to be at AECOM. With accelerating infrastructure investment worldwide, our services are in great demand. We invite you to bring your bold ideas and big dreams and become part of a global team of over 50,000 planners, designers, engineers, scientists, digital innovators, program and construction managers and other professionals delivering projects that create a positive and tangible impact around the world. We're one global team driven by our common purpose to deliver a better world. Join us. jobDescription In AECOM’s AI Engineering team your code will directly shape the physical world around us. We build AI-driven technology that revolutionises how infrastructure and buildings are designed and engineered; reducing waste, cutting CO₂, and making the built environment more efficient and sustainable. This is where software has measurable, real-world impact. With our AI Engineering team we’ve created a unique setup: a lean, highly technical team with the speed and ownership of a start up, backed by the scale, resources, and domain expertise of one of the world’s leading engineering firms. There has never been a better time to be at AECOM. We are leading the industry’s AI transformation, and with our people and technology we deliver excellence and innovate with impact. We invite you to bring your bold ideas and big dreams to solve the world’s most complex challenges. We're one global team driven by our common purpose to deliver a better world. Join us. What You’ll Do As part of the Assurance & Insight team, you will focus on ensuring the quality and reliability of our AI-driven systems. This role sits at the intersection of testing, AI evaluation, and operational quality. You will help build the foundations for how we test, benchmark, and monitor both software systems and LLM-driven outputs ensuring that what we deploy performs as expected in real-world scenarios. You will take ownership of quality across our AI systems, from automated testing to LLM evaluation and production monitoring. Your work will help reduce regressions, improve reliability, and free up engineering time to focus on building. We're not looking for years of experience. We're looking for someone detail-oriented who wants to own a domain, can write clean Python, and is genuinely curious about how AI systems behave. It's a good fit for a recent graduate who wants end-to-end ownership early in their career, is comfortable working close to AI systems, and cares about making quality measurable rather than subjective. qualifications Must-Have Qualifications Degree in computer science, software engineering, or a related technical field or equivalent demonstrable experience Exposure to LLMs or AI systems (even personal projects or coursework counts) Familiarity with test frameworks such as pytest or similar Understanding of CI/CD concepts and release pipelines Comfortable working with Python and APIs Experience writing or evaluating prompts for language models Strong attention to detail and a methodical approach to problem-solving Preferred Skills Exposure to AI/LLM tools or projects Experience with testing, QA, or automation (academic or personal projects) Interest in data quality, monitoring, or analytics Experience working in team-based or project-based environments additionalInformation Our Hiring Process 25-minute screening call Take-home challenge: A hands-on task to assess your problem-solving and technical skills Combined technical and cultural interview (in-person) Whiteboard Interview: 1-hour with 2 of our engineers to discuss your solution to the take-home challenge Culture fit: 30-minute meeting with our leadership team Why Join Us? Work on real-world problems where AI creates measurable impact. Be part of a team where your work matters, and your ideas become real. Collaborate with sharp, driven colleagues in a culture of trust, ownership, and high standards. Contribute to making the built environment smarter and more sustainable.