TT | EAD | Analyst | Automation Tester | Mumbai | Engineering
Location: Mumbai
Your work profile
As an Analyst in AI-driven Quality Engineering, you will play a critical role in validating the reliability, accuracy, and ethical behavior of Large Language Model (LLM)-based applications. You will work at the intersection of AI, testing, and business workflows to ensure that AI systems are trustworthy, performant, and production-ready.
Key Responsibilities
1. LLM Functional Testing
- Design and execute test scenarios for LLM-based applications (chatbots, copilots, generative AI systems)
- Validate response correctness, relevance, and contextual understanding
- Perform prompt-based testing to assess model behaviour under varied inputs
2. Prompt Engineering Validation
- Create, refine, and optimize prompts to test different use cases and edge conditions
- Identify prompt vulnerabilities such as hallucinations, bias, and unsafe outputs
- Evaluate consistency and determinism across multiple runs
3. AI Quality Dimensions Validation
- Validate outputs across key AI quality pillars: Accuracy relevance
- Coherence fluency
- Bias fairness
- Safety compliance
- Perform adversarial testing to uncover unexpected or harmful responses
4. Test Data Scenario Design
- Develop structured and unstructured datasets for AI validation
- Simulate real-world user interactions across domains (banking, telecom, healthcare, etc.)
- Design boundary, negative, and stress scenarios for LLMs
5. Automation Tooling
- Leverage automation tools/frameworks for AI testing (Python, APIs, prompt libraries)
- Use AI testing platforms (Botium, LangChain testing tools, Azure AI Studio)
- Integrate tests into CI/CD pipelines for continuous validation
6. Performance Scalability Testing
- Validate response latency, throughput, and system behaviour under load
- Assess performance of LLM APIs and integrated AI workflows
7. Governance, Risk Compliance
- Ensure adherence to Responsible AI principles
- Validate compliance with data privacy, security, and regulatory standards
- Document test evidence for audit and governance reviews
8. Defect Analysis Reporting
- Identify, log, and track AI-specific defects (hallucination, drift, inconsistency)
- Provide actionable insights and recommendations for model tuning
Required Skills Qualifications
Core Skills
- Understanding of software testing fundamentals (STLC, defect lifecycle)
- Basic knowledge of AI/ML and LLM concepts (prompting, tokens, embeddings)
- Familiarity with API testing (Postman / REST clients)
- Exposure to automation (Python / Selenium / Playwright basic level)
Good to Have
- Experience with AI/LLM tools (OpenAI, Azure OpenAI, Hugging Face)
- Knowledge of prompt engineering techniques
- Understanding of NLP concepts
- Exposure to tools like Botium, LangChain, or similar AI testing frameworks
Behavioral Competencies
- Analytical thinking with strong attention to detail
- Curiosity to explore AI behavior and edge cases
- Strong communication skills for articulating AI risks and findings
- Ability to work in agile and fast-evolving environments
Disclaimer : This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.
1-6 years96
Get Personalized Job Matches with 1 Click
Job Description
Download ResumeTT | EAD | Analyst | Automation Tester | Mumbai | Engineering
Location: Mumbai
Your work profile
As an Analyst in AI-driven Quality Engineering, you will play a critical role in validating the reliability, accuracy, and ethical behavior of Large Language Model (LLM)-based applications. You will work at the intersection of AI, testing, and business workflows to ensure that AI systems are trustworthy, performant, and production-ready.
Key Responsibilities
1. LLM Functional Testing
- Design and execute test scenarios for LLM-based applications (chatbots, copilots, generative AI systems)
- Validate response correctness, relevance, and contextual understanding
- Perform prompt-based testing to assess model behaviour under varied inputs
2. Prompt Engineering Validation
- Create, refine, and optimize prompts to test different use cases and edge conditions
- Identify prompt vulnerabilities such as hallucinations, bias, and unsafe outputs
- Evaluate consistency and determinism across multiple runs
3. AI Quality Dimensions Validation
- Validate outputs across key AI quality pillars: Accuracy relevance
- Coherence fluency
- Bias fairness
- Safety compliance
- Perform adversarial testing to uncover unexpected or harmful responses
4. Test Data Scenario Design
- Develop structured and unstructured datasets for AI validation
- Simulate real-world user interactions across domains (banking, telecom, healthcare, etc.)
- Design boundary, negative, and stress scenarios for LLMs
5. Automation Tooling
- Leverage automation tools/frameworks for AI testing (Python, APIs, prompt libraries)
- Use AI testing platforms (Botium, LangChain testing tools, Azure AI Studio)
- Integrate tests into CI/CD pipelines for continuous validation
6. Performance Scalability Testing
- Validate response latency, throughput, and system behaviour under load
- Assess performance of LLM APIs and integrated AI workflows
7. Governance, Risk Compliance
- Ensure adherence to Responsible AI principles
- Validate compliance with data privacy, security, and regulatory standards
- Document test evidence for audit and governance reviews
8. Defect Analysis Reporting
- Identify, log, and track AI-specific defects (hallucination, drift, inconsistency)
- Provide actionable insights and recommendations for model tuning
Required Skills Qualifications
Core Skills
- Understanding of software testing fundamentals (STLC, defect lifecycle)
- Basic knowledge of AI/ML and LLM concepts (prompting, tokens, embeddings)
- Familiarity with API testing (Postman / REST clients)
- Exposure to automation (Python / Selenium / Playwright basic level)
Good to Have
- Experience with AI/LLM tools (OpenAI, Azure OpenAI, Hugging Face)
- Knowledge of prompt engineering techniques
- Understanding of NLP concepts
- Exposure to tools like Botium, LangChain, or similar AI testing frameworks
Behavioral Competencies
- Analytical thinking with strong attention to detail
- Curiosity to explore AI behavior and edge cases
- Strong communication skills for articulating AI risks and findings
- Ability to work in agile and fast-evolving environments
Disclaimer : This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.