Loading...
Agentic AI Test Engineer
Comcast
On-siteseniorPosted May 6, 2026
About the Role
Design and implement automated evaluation frameworks for AI Agents using LLM-as-a-judge patterns, perform hands-on validation of AI Agents, and build/maintain full-stack automation across web and API environments.
Requirements
Requires 5+ years as an SDET/Automation Engineer with expertise in Python, building custom automation frameworks, and a solid understanding of AI Agent evaluation techniques using LLMs as a judge.
Full Job Description
Make your mark at Comcast -- a Fortune 30 global media and technology company. From the connectivity and platforms we provide, to the content and experiences we create, we reach hundreds of millions of customers, viewers, and guests worldwide. Become part of our award-winning technology team that turns big ideas into cutting-edge products, platforms, and solutions that our customers love. We create space to innovate, and we recognize, reward, and invest in your ideas, while ensuring you can proudly bring your authentic self to the workplace. Join us. You’ll do the best work of your career right here at Comcast. (In most cases, Comcast prefers to have employees on-site collaborating unless the team has been designated as virtual due to the nature of their work. If a position is listed with both office locations and virtual offerings, Comcast may be willing to consider candidates who live greater than 100 miles from the office for the remote option.)
We are building the next generation of our chat experience, transitioning from a human-assisted workflow to a fully agentic, AI-driven system. As an AI Agentic Software Engineer, you will create re-usable AI agents that will be used across Xfinity applications. Your initial focus will be to enable agentic capabilities at our Xfinity Assistant (chat application) with a strong passion for customer experience and a commitment to building intuitive, high-quality interactions.
Job Summary
The AI Agentic Team designs, builds, and runs intelligent agents end to end. We own agent logic, workflows, integrations, production deployment, and ongoing performance. By combining build and operations, we deliver agentic systems that are reliable, secure, and scalable in real-world use.We are building the next generation of our chat experience, transitioning from a human-assisted workflow to a fully agentic, AI-driven system. As an AI Agentic Software Engineer, you will create re-usable AI agents that will be used across Xfinity applications. Your initial focus will be to enable agentic capabilities at our Xfinity Assistant (chat application) with a strong passion for customer experience and a commitment to building intuitive, high-quality interactions.
Job Description
We are seeking a AI Agentic Test Engineer to join our core automation team. This role is designed for a hands-on engineer who is passionate about the intersection of AI and Quality Engineering. You will focus on building automated evaluation loops for AI Agents using LLM-as-a-judge patterns and maintaining robust Web and API test suites. The ideal candidate is a Python expert who is also comfortable working across various modern tech stacks and CI/CD environments.
AI & Agent Evaluation (Primary Focus)
LLM-as-a-Judge: Design and implement automated evaluation frameworks that utilize Large Language Models to assess the quality, accuracy, and safety of Agent responses.
Agentic Testing: Perform hands-on validation of AI Agents within complex environments, moving beyond static assertions to intelligent, semantic validation.
Data Validation: Automate the testing of real-time data streams and LLM outputs to ensure consistency and prevent regression in Agent behavior.
Full-Stack Automation & Coding
Python Mastery: Expert in Python for building custom test utilities, framework extensions, and automation scripts.
Cross-Functional Tooling: Develop and maintain automation across TypeScript, Java, and React environments to support various application layers.
Web & API Automation: Build high-performance automated tests for modern web interfaces and RESTful APIs, ensuring end-to-end system reliability.
Internal Tooling: Help build or maintain internal quality dashboards and tools using frameworks like Streamlit.
CI/CD & Technical Operations
Pipeline Execution: Integrate automated tests into CI/CD pipelines (e.g., Jenkins, Concourse) to ensure continuous quality feedback throughout the development lifecycle.
Code Quality: Actively participate in technical code reviews, focusing on script efficiency, maintainability, and clean code principles.
Troubleshooting: Use logs and observability tools to perform deep-dive root cause analysis on automation failures and system defects.
What You’ll Need:
5+ years of experience as an SDET or Automation Engineer.
Strong technical background in building and maintaining custom automation frameworks.
Ability to work independently on complex coding tasks while collaborating effectively in an Agile environment.
Core Language: Expert-level hands-on coding in Python.
AI/LLM Skills: Solid understanding of Agent Evaluation techniques and using LLMs as a judge.
Automation Breadth: Proven experience in Web and API automation.
Multi-Stack Knowledge (Good to Have):
Languages: TypeScript, Java.
Frameworks: React, Streamlit.
DevOps: CI/CD pipeline management (Jenkins/Concourse).
Disclaimer: This information has been designed to indicate the general nature and level of work performed by employees in this role. It is not designed to contain or be interpreted as a comprehensive inventory of all duties, responsibilities and qualifications.
Skills
Adaptability, Agentic AI, AI Agent Orchestration, AI Agents, Automation, Critical Thinking, Python (Programming Language)Compensation
Primary Location Pay Range: $114,985.89 - $172,478.83Comcast intends to offer the selected candidate base pay within this range, dependent on job-related, non-discriminatory factors such as experience. The application window is 30 days from the date job is posted, unless the number of applicants requires it to close sooner or later.Base pay is one part of the Total Rewards that Comcast provides to compensate and recognize employees for their work. Most sales positions are eligible for a Commission under the terms of an applicable plan, while most non-sales positions are eligible for a Bonus. Additionally, Comcast provides best-in-class Benefits to eligible employees. We believe that benefits should connect you to the support you need when it matters most, and should help you care for those who matter most. That’s why we provide an array of options, expert guidance and always-on tools, that are personalized to meet the needs of your reality – to help support you physically, financially and emotionally through the big milestones and in your everyday life. Please visit the compensation and benefits summary on our careers site for more details.
Education
Bachelor's DegreeWhile possessing the stated degree is preferred, Comcast also may consider applicants who hold some combination of coursework and experience, or who have extensive related professional experience.Certifications (if applicable)
Relevant Work Experience
5-7 YearsComcast is an equal opportunity workplace. We will consider all qualified applicants for employment without regard to race, color, religion, age, sex, sexual orientation, gender identity, national origin, disability, veteran status, genetic information, or any other basis protected by applicable law.