Jobs › Vizient

Lead, AI Quality & Reliability Engineering

Vizient · Chicago, IL · $88,900–$155,500 a year · Full time

Posted Sep 21, 2026 · We last checked this listing on Oct 5, 2026

Apply at Vizient

Likely interview questions for this role

Written from this job description, not a generic list. Each one notes what the interviewer is really checking.

About your past

Tell me about a time you led a QA team through a big change in tools or testing approach.

What they are checking: Leading people through modernization while still delivering, since this role mixes team leadership and hands-on work.

Tell me about a defect that escaped to production on your watch. What did you change afterward?

What they are checking: Ownership and whether they fix the process gap rather than only the single bug.

How do you explain AI quality results to business leaders who just want to know if it works?

What they are checking: Translating technical evaluation metrics into clear risk statements that non-engineers can act on.

What would you do

A product team wants to ship an AI feature that supports a healthcare workflow, but your evaluation shows a 4 percent error rate. What do you do?

What they are checking: Judging acceptable risk in context, and communicating it clearly so the business makes an informed call.

How would you build a validation dataset for a healthcare workflow when real patient data is restricted?

What they are checking: Balancing realistic test coverage with privacy rules, using synthetic or de-identified data responsibly.

Several AI initiatives need your team at the same time. How do you decide where quality engineers go?

What they are checking: Risk-based prioritization and cross-functional coordination when demand outstrips the team's capacity.

About the work

How do you test an LLM-enabled workflow when the same input can give different outputs each time?

What they are checking: Whether they know evaluation approaches like rubrics, golden datasets and tolerance thresholds instead of exact-match assertions.

Walk me through the quality strategy you would set up for an agentic system that calls tools and takes multi-step actions.

What they are checking: Thinking about tool-call correctness, guardrails, failure recovery and tracing, not just checking the final answer.

What runtime quality controls would you put in place once an AI feature is in production?

What they are checking: Whether they extend quality past release into monitoring, drift detection and alerting, which is core to reliability.

How have you used AI to speed up testing itself, like generating test cases or test data? Where did it fall short?

What they are checking: Hands-on experience with AI-assisted testing and honest awareness of its limits.

Practice this interview out loud.

Offer builds a real interview for this exact role at Vizient from your resume and this job description, asks the questions one at a time, and tells you what landed. The first one is free.

Practice this out loud

The full job description

As published by Vizient.

When you’re the best, we’re the best. We instill an environment where employees feel engaged, satisfied and able to contribute their unique skills and talents while living and working as their authentic selves . We provide extensive opportunities for personal and professional development, building both employee competence and organizational capability to fuel exceptional performance through an inclusive environment both now and in the future. Summary In this role, you will lead the day-to-day execution and delivery of AI Quality Engineering initiatives supporting Vizient's enterprise AI transformation efforts. You will help ehelp implement AI Quality Engineering practices, AI validation processes, AI-assisted testing approaches, runtime quality controls, and scalable testing frameworks supporting responsible deployment of AI-powered business solutions. This role combines hands-on quality engineering leadership, AI-enabled testing modernization, healthcare workflow validation, team leadership, and cross-functional delivery coordination to ensure the reliability, performance, and governance of AI-enabled systems. Responsibilities • Lead day-to-day AI Quality Engineering activities supporting AI-powered applications, LLM-enabled workflows, intelligent automation solutions, agentic systems, and enterprise AI platforms. • Implement AI Quality Engineering practices, including AI-native testing approaches, validation processes, runtime quality controls, reusable testing accelerators, and scalable testing workflows. • Coordinate AI validation efforts, including functional testing, prompt testing, workflow validation, regression testing, defect management, release readiness, and production quality assurance. • Partner with AI Engineering, AIOps, LLMOps, Security, Governance, Clinical, Data, and Product teams to ensure scalable and effective quality engineering processes across AI initiatives. • Support runtime quality and reliability through observability, monitoring, telemetry analysis, incident support, drift detection, distributed tracing, and continuous improvement initiatives. • Drive adoption of AI-assisted testing, intelligent automation, reusable test assets, and modern quality engineering practices that improve efficiency and quality outcomes. • Lead delivery coordination activities including sprint execution, testing planning, issue tracking, risk identification, dependency management, and release support. • Collaborate with clinical, operational, and engineering stakeholders to validate healthcare workflows, payer operations, and AI-enabled business processes while supporting responsible AI deployment. • Mentor quality engineers, analysts, contractors, and delivery teams while fostering a culture of continuous learning, engineering excellence, and operational accountability. Qualifications • Bachelor's degree in Computer Science, Information Systems, Engineering, or related field preferred. • 7 or more years of experience in Quality Engineering, Software Testing, Quality Assurance, or related technology disciplines required. • 1 or more years of experience supporting AI Quality Engineering, AI testing, AI validation, AI-enabled automation, or modern AI engineering initiatives preferred. • Strong experience supporting Quality Engineering or Quality Assurance activities across enterprise platforms, APIs, operational workflows, healthcare applications, or integrated business systems required. • Experience with enterprise testing practices, test automation frameworks, SDLC methodologies, Agile delivery models, defect management, release validation, and production support coordination required. • Strong analytical, organizational, communication, collaboration, and leadership skills required. • Ability to manage multiple priorities within fast-paced, evolving, and highly collaborative enterprise environments required. • Experience with Python, APIs, automation frameworks, AI-enabled workflows, or modern software engineering practices preferred. • Experience supporting AI/ML platforms, orchestration frameworks, runtime monitoring, observability tooling, or AI operational support processes preferred. • Familiarity with AI-assisted testing, AI evaluation approaches, prompt testing, runtime observability, operational monitoring, drift detection, or AI operational assurance practices preferred. • Experience supporting enterprise modernization or transformation initiatives involving AI, automation, or operational scaling preferred. #LI-JB1 Estimated Hiring Range: At Vizient, we consider skills, experience, and organizational needs in our compensation approach. Geographic factors may adjust the range estimate and hires typically fall below the top range. Compensation decisions are tailored to individual circumstances. The current salary range for this role is $88,900.00 to $155,500.00. This position is also incentive eligible. Vizient has a comprehensive benefits plan! Please view our benefits here: http://www.vizientinc.com/about-us/careers Equal Opportunity Employer: Females/Minorities/Veterans/Individuals with Disabilities The Company is committed to equal employment opportunity to all employees and applicants without regard to race, religion, color, gender identity, ethnicity, age, national origin, sexual orientation, disability status, veteran status or any other category protected by applicable law.

Apply at Vizient