Gen AI Data Engineer at NielsenIQ in Pune, Maharashtra, India

companyDescription NIQ is the world’s leading consumer intelligence company, delivering the most complete understanding of consumer buying behavior and revealing new pathways to growth. In 2023, NIQ combined with GfK, bringing together the two industry leaders with unparalleled global reach. With a holistic retail read and the most comprehensive consumer insights—delivered with advanced analytics through state-of-the-art platforms—NIQ delivers the Full View™. NIQ is an Advent International portfolio company with operations in 100+ markets, covering more than 90% of the world’s population. jobDescription As a GenAI Data Engineer, you will play a critical role in building the data foundation that powers generative AI applications. You will work closely with software engineers, ML engineers, and product leadership to enable rapid experimentation and drive scalable, production-grade GenAI systems. You will be responsible for designing flexible and scalable data pipelines, enabling retrieval systems, and ensuring high-quality data flows across the GenAI lifecycle—from experimentation to production deployment. Core Responsibilities Data Engineering & Pipeline Development Design and build scalable batch and near real-time data pipelines using modern data processing frameworks. Develop robust ETL/ELT workflows for ingesting and transforming structured and unstructured data (documents, PDFs, APIs, logs, etc.). Leverage cloud-native orchestration and data processing solutions to ensure reliability and scalability. Implement reusable data frameworks that support rapid experimentation and iteration cycles. Ensure data quality through validation, schema enforcement, and automated checks. GenAI Data Preparation & Experimentation Prepare and curate datasets for GenAI use cases including RAG, embeddings, and fine-tuning workflows. Implement data processing steps like chunking, tokenization, metadata enrichment, and semantic structuring. Enable fast experimentation loops by supporting dynamic datasets and evaluation pipelines. Collaborate with engineers and product teams to iterate quickly on features and experiments. Transition experimental pipelines into production-ready, robust workflows. Vector Databases & Retrieval Systems Build and maintain embedding pipelines using LLM providers and open-source models. Design and optimize retrieval systems using cloud-native vector databases and hybrid storage solutions. Work with relational databases supporting vector capabilities (e.g., PostgreSQL with vector extensions). Implement and optimize RAG pipelines, including indexing, retrieval, ranking, and refresh strategies. Manage lifecycle of embeddings, vector indexes, and retrieval datasets. Data Storage & Platform Engineering Work with cloud-native data platforms and storage solutions (data lakes, lakehouses, object storage). Design efficient storage schemas for both analytical and retrieval workloads. Optimize relational and hybrid data stores for low-latency, high-throughput access patterns. Ensure cost-effective and scalable data storage strategies. Productionization & Scalability Convert experimental workflows into scalable, reliable production pipelines. Optimize pipelines for performance, cost, and reliability. Implement incremental processing, caching, and efficient refresh strategies. Monitoring & Data Observability Implement monitoring for pipeline health, data freshness, and quality. Track dataset drift, embedding drift, and retrieval effectiveness. Build logging, alerting, and observability frameworks for data systems. Collaboration & Cross-Functional Work Partner closely with engineering teams, product leadership, and data scientists to define and deliver data solutions. Act as a bridge between rapid experimentation and production engineering. Contribute to architecture decisions and GenAI data best practices. Document pipelines, architectures, and data models clearly. Nice-to-Have / Growth Areas Experience with GenAI frameworks (e.g., LangChain, LlamaIndex, or similar) Exposure to knowledge graphs and graph-based retrieval approaches Understanding of data governance, lineage, and cataloging Experiment tracking and dataset versioning Experience working with multi-modal datasets (text, image, audio) qualifications 2-3 years of experience in Data Engineering or related roles. Strong proficiency in Python and SQL. Hands-on experience with modern data processing frameworks and orchestration tools. Experience working with cloud-native data platforms (Azure, AWS, or GCP). Experience with relational databases such as PostgreSQL, including extensions for advanced workloads (e.g., vector storage). Strong understanding of building scalable data pipelines for both structured and unstructured data. Familiarity with GenAI concepts such as LLMs, embeddings, and RAG architectures. Soft Skills Strong collaborator comfortable working with engineers, product managers, and leadership. Ability to balance rapid experimentation with production rigor. Strong problem-solving and debugging capabilities across data systems. Clear communicator with strong documentation practices. Adaptable and thrives in fast-moving, GenAI-driven environments. additionalInformation Enjoy a flexible and rewarding work environment with peer-to-peer recognition platforms. Recharge and revitalize with help of wellness plans made for you and your family. Plan your future with financial wellness tools. Stay relevant and upskill yourself with career development opportunities. Our Benefits Flexible working environment Volunteer time off LinkedIn Learning Employee-Assistance-Program (EAP) NIQ may utilize artificial intelligence (AI) tools at various stages of the recruitment process, including résumé screening, candidate assessments, interview scheduling, job matching, communication support, and certain administrative tasks that help streamline workflows. These tools are intended to improve efficiency and support fair and consistent evaluation based on job-related criteria. All use of AI is governed by NIQ’s principles of fairness, transparency, human oversight, and inclusion. Final hiring decisions are made exclusively by humans. NIQ regularly reviews its AI tools to help mitigate bias and ensure compliance with applicable laws and regulations. If you have questions, require accommodations, or wish to request human review were permitted by law, please contact your local HR representative. For more information, please visit NIQ’s AI Safety Policies and Guiding Principles: https://www.nielseniq.com/global/en/ai-safety-policies. About NIQ NIQ is the world’s leading consumer intelligence company, delivering the most complete understanding of consumer buying behavior and revealing new pathways to growth. In 2023, NIQ combined with GfK, bringing together the two industry leaders with unparalleled global reach. With a holistic retail read and the most comprehensive consumer insights—delivered with advanced analytics through state-of-the-art platforms—NIQ delivers the Full View™. NIQ is an Advent International portfolio company with operations in 100+ markets, covering more than 90% of the world’s population. For more information, visit NIQ.com Want to keep up with our latest updates? Follow us on: LinkedIn | Instagram | Twitter | Facebook Our commitment to Diversity, Equity, and Inclusion At NIQ, we are steadfast in our commitment to fostering an inclusive workplace that mirrors the rich diversity of the communities and markets we serve. We believe that embracing a wide range of perspectives drives innovation and excellence.  All employment decisions at NIQ are made without regard to race, color, religion, sex (including pregnancy, sexual orientation, or gender identity), national origin, age, disability, genetic information, marital status, veteran status, or any other characteristic protected by applicable laws. We invite individuals who share our dedication to inclusivity and equity to join us in making a meaningful impact. To learn more about our ongoing efforts in diversity and inclusion, please visit the https://nielseniq.com/global/en/news-center/diversity-inclusion