Jobs › Vizient

QA Automation Engineer – Agentic AI, UI, and API

Vizient · Chicago, IL · $102,400–$179,000 a year · Full time

Posted Sep 21, 2026 · We last checked this listing on Oct 5, 2026

Apply at Vizient

Likely interview questions for this role

Written from this job description, not a generic list. Each one notes what the interviewer is really checking.

About your past

Tell me about a serious bug you caught before release. How did you find it?

What they are checking: Curiosity and thoroughness, the instincts that make a strong quality engineer.

What would you do

Production monitoring shows an AI feature's answers drifting over a few weeks. What do you do?

What they are checking: Using telemetry and tracing to investigate drift and feed findings back into testing.

A developer says your AI test is too strict and is blocking release. How do you respond?

What they are checking: Defending quality standards with evidence while staying open to adjusting thresholds fairly.

About the work

How would you test an agentic AI system that decides on its own which tools to call?

What they are checking: Thinking about tool selection accuracy, step tracing and failure paths, not just the final answer.

What's your approach to prompt testing and regression when a prompt or model version changes?

What they are checking: Building evaluation datasets and regression checks that catch quality drops from AI changes.

Walk me through a UI and API automation framework you built. What made it maintainable?

What they are checking: Solid traditional automation skill, since the role covers UI and API alongside AI.

How do you handle flaky tests caused by nondeterministic AI outputs?

What they are checking: Designing tolerant assertions, scoring and thresholds instead of disabling tests or ignoring failures.

When does human-in-the-loop review belong in an AI testing process?

What they are checking: Judgment about where automated checks fall short, especially in healthcare workflows.

How have you used AI to help write or maintain tests? What were the limits?

What they are checking: Hands-on AI-assisted testing experience with honest awareness of where it fails.

How do you integrate AI validation into a CI/CD pipeline without making builds too slow or costly?

What they are checking: Practical pipeline design that balances coverage, run time and model call costs.

Practice this interview out loud.

Offer builds a real interview for this exact role at Vizient from your resume and this job description, asks the questions one at a time, and tells you what landed. The first one is free.

Practice this out loud

The full job description

As published by Vizient.

When you’re the best, we’re the best. We instill an environment where employees feel engaged, satisfied and able to contribute their unique skills and talents while living and working as their authentic selves . We provide extensive opportunities for personal and professional development, building both employee competence and organizational capability to fuel exceptional performance through an inclusive environment both now and in the future. Responsibilities: • Design and execute quality engineering strategies for AI-powered applications, agentic AI systems, user interfaces, APIs, LLM-enabled workflows, intelligent automation, and integrated enterprise platforms. • Develop scalable test automation frameworks, reusable testing accelerators, validation processes, and automated workflows that improve quality, efficiency, and test coverage. • Implement AI validation approaches, including functional, prompt, workflow, integration, regression, and human-in-the-loop testing, along with validation datasets and quality evaluation methods. • Advance AI-assisted testing and modern quality engineering practices that address nondeterministic AI behavior, intelligent orchestration, and evolving healthcare workflows. • Support runtime quality and reliability through telemetry, distributed tracing, observability, monitoring, drift detection, incident analysis, and production validation. • Collaborate with engineering, AIOps, LLMOps, security, governance, data, clinical, and operational clients to integrate quality practices throughout the software and AI development lifecycle. • Coordinate test execution, defect management, release validation, risk identification, and production support activities across AI-enabled solutions. • Evaluate and implement testing, automation, observability, and AI quality engineering tools and technologies that improve scalability, reliability, and traceability. • Mentor quality engineers and analysts by providing technical guidance, sharing automation practices, and fostering continuous learning and engineering discipline. • Communicate quality risks, technical findings, implementation dependencies, and recommendations to technical and business clients while promoting continuous improvement and responsible AI adoption. Estimated Hiring Range: At Vizient, we consider skills, experience, and organizational needs in our compensation approach. Geographic factors may adjust the range estimate and hires typically fall below the top range. Compensation decisions are tailored to individual circumstances. The current salary range for this role is $102,400.00 to $179,000.00. This position is also incentive eligible. Vizient has a comprehensive benefits plan! Please view our benefits here: http://www.vizientinc.com/about-us/careers Equal Opportunity Employer: Females/Minorities/Veterans/Individuals with Disabilities The Company is committed to equal employment opportunity to all employees and applicants without regard to race, religion, color, gender identity, ethnicity, age, national origin, sexual orientation, disability status, veteran status or any other category protected by applicable law.

Apply at Vizient