Annotation QA Analyst - Content Platform at Spotify in New York, NY

Summary: Spotify is a leading audio streaming service that designs consumer experiences across various platforms. They are seeking an Annotation QA Analyst to ensure the quality of annotated datasets used by Content Platform teams, focusing on machine learning and AI-driven development. Responsibilities: - Review annotated data to ensure it meets Spotify’s quality standards and policies, prioritizing work based on business needs - Deliver high-quality, timely results for Product and Engineering teams using established QA frameworks and metrics such as agreement rates and consensus - Handle complex edge cases, helping define ground truth and reduce ambiguity across datasets - Identify patterns, insights, and areas for improvement, and communicate findings clearly to both technical and non-technical partners - Contribute to feedback loops between annotation teams, R&D collaborators, and content policy experts to improve workflows and outputs - Help develop and refine annotation guidelines, supporting annotator training and continuous improvement - Collaborate closely with teammates across multiple projects and domains Required Qualifications: - You have experience working with annotation, data quality, or QA processes in ML/AI environments - You're familiar with LLM or AI-driven annotation workflows and human-in-the-loop systems - You're comfortable reviewing large-scale datasets across different modalities such as text, audio, images, or video - You care about quality and consistency, and bring a structured approach to evaluating data - You communicate clearly and can explain complex ideas in a simple, accessible way - You collaborate well with cross-functional partners in fast-moving environments - You have a solid understanding of the machine learning lifecycle, from data collection to deployment - You're comfortable working with emerging AI tools and agent workflows - You bring curiosity and interest in music, podcasts, or audiobooks Preferred Qualifications: - You have familiarity with SQL or music metadata standards (nice to have) Required Skills: Annotation, Data Quality, Quality Assurance, Machine Learning Lifecycle, LLM Annotation Workflows, Human-in-the-Loop Systems, Multimodal Data Review, SQL Benefits: Health insurance, Six month paid parental leave, 401(k) retirement plan, Monthly meal allowance, 23 paid days off, 13 paid flexible holidays