Quality Engineering Specialist

Quality Engineer Specialist – AI & LLM Solutions

We are looking for a QA Engineer Specialist specializing in AI and LLM-based systems to ensure the quality, reliability, and performance of AI products used across global media and marketing platforms. The role focuses on testing LLMs, RAG pipelines, AI agents, and ML/data workflows, with a strong emphasis on automation and AI-specific quality metrics.

Key Responsibilities

• Own quality assurance for AI products, including LLMs, RAG systems, agents, and ML pipelines.
• Design and execute AI-specific test strategies (factuality, hallucinations, robustness, safety).
• Validate agent behaviors, AI outputs, and performance (latency, stability, quality).
• Build and maintain automated test suites integrated into CI/CD pipelines.
• Prevent regressions and assess the impact of model, embedding, or prompt changes.
• Collaborate with engineers and Product teams to challenge acceptance criteria and system robustness.

Required Skills & Experience

• 7-9 years of experience in QA or Test Engineering (exposure to AI/ML systems preferred).
• Strong experience with test automation in Python.
• Hands-on experience with API testing and end‑to‑end testing frameworks.
• Working understanding of LLMs and RAG concepts (embeddings, vector search).
• Experience integrating tests into CI/CD pipelines.
• Basic experience with Docker and test environments.

Nice to Have

• Experience with AI/LLM evaluation techniques (factuality, toxicity, consistency).
• Exposure to Azure OpenAI or Vertex AI platforms.
• Performance or load testing experience.

Why Join us

• Play a key role in defining AI quality and trust for enterprise GenAI solutions.
• Work on production AI platforms used by global media brands.
• Be part of a growing Media Tech Global Capability Center in India.