Software Engineer in Test - AI

Location: Salt Lake City, UT

In this role, you will build automation, evaluate AI-driven behavior, validate responsible AI controls, and identify risks early so our products are reliable, scalable, production-ready, and trusted by millions of users.

Key Responsibilities

• Become a Subject Matter Expert on the AI platform and possess a deep understanding of system interactions, upstream/downstream dependencies, and data flows.
• Design, develop, and maintain automated test frameworks for AI platform services, APIs, agents, prompt-based workflows, and RAG-enabled applications.
• Develop automated AI evaluation suites and regression benchmarks that continuously measure model behavior and detect quality degradation before release.
• Build and execute functional, integration, end-to-end, regression, performance, and reliability tests for AI-driven systems and services.
• Define evaluation strategies, quality metrics, and acceptance criteria for AI-generated outputs, including accuracy, relevance, consistency, grounding, safety, and business value.
• Validate responsible AI controls, permissions, guardrails, data handling practices, and business rules that protect customer trust.
• Partner with Engineering, Product, and Support throughout the software lifecycle to drive risk-based testing strategies, influence quality to ensure production readiness.
• Integrate automated testing, AI evaluations, and release validation into CI/CD pipelines.
• Investigate, triage, and communicate defects, quality issues, and production incidents, driving root cause analysis and continuous improvement.
• Utilize observability tools, logs, metrics, and traces to investigate production issues and perform root cause analysis.
• Establish and promote testing standards, automation patterns, and AI quality practices that improve reliability and delivery speed.

Required Qualifications

• 3+ years of experience in software testing, quality engineering, test automation, or software development.
• Hands-on experience developing automated tests, testing tools, or quality frameworks using Python, Selenium and Playwright.
• Experience testing APIs, microservices, distributed systems, backend services, or event-driven architectures.
• Strong experience testing AI-enabled applications using technologies such as LLMs, LangChain, LangGraph, or similar platforms.
• Hands-on experience evaluating AI-generated outputs using datasets, scoring rubrics, golden test sets, benchmarking frameworks, or automated quality checks.
• Experience testing RAG systems, including retrieval quality, embeddings, grounding, citations, context accuracy, and streaming responses.
• Experience working with SQL and NoSQL technologies, including relational, document, key-value, or vector databases.
• Experience integrating automated testing into CI/CD pipelines and modern software delivery practices.
• Experience investigating production issues and using incident and defect data to improve system quality and reliability.
• Strong understanding of test automation, performance testing, risk-based testing, release validation, and responsible AI principles.
• Excellent collaboration and communication skills, with the ability to clearly articulate quality risks and acceptance criteria.

Preferred Qualifications

• Experience with AWS, Kubernetes, Docker, and cloud-native architectures.
• Experience testing event-driven systems using Kafka or similar messaging platforms.
• Familiarity with application security testing and OWASP Top 10 principles.
• Proficiency in Python and at least one additional programming language such as Java, Ruby, or Go.
• Experience with reliability engineering practices, including observability, production readiness reviews, incident analysis, SLOs, and continuous improvement.
• Experience conducting performance, load, stress, scalability, or resilience testing.