Get introduced to the
most exciting companies in
our portfolio.

About the Role

Hello Patient is seeking a Conversational Quality Analyst to ensure Mia—our AI assistant for medical practices—performs reliably and safely before going live with customers. Mia supports voice, text, and chat communications by answering patient calls, scheduling appointments, handling prescription refill workflows, managing referrals, and routing urgent issues—so front desk teams can focus on patients in front of them instead of being tied to the phone.

In this role, you’ll review high-volume simulated conversations to uncover failure modes, define clear evaluation rules that describe what “good” looks like, and maintain those rules as workflows, customers, and product behavior evolve.

This position is a critical quality gate and the last line of defense before launch—your judgment helps determine whether issues are caught pre-release or discovered by customers in production. Your work directly impacts patient experience, customer trust, and operational outcomes across urgent care clinics, dental practices, veterinary offices, and other healthcare organizations that rely on Mia every day.

Responsibilities

• Design and maintain evaluation rules: Define clear, testable pass/fail criteria for Mia’s conversational behavior by translating customer workflows and success metrics into evaluation standards. Apply leadership-defined severity frameworks to prioritize what matters most.

• Build and manage test suites: Create structured test suites to proactively surface failure modes before launch. Organize tests by risk category and document explicit success criteria so quality can be validated consistently.

• Review simulations and make launch decisions: Review 100+ simulated conversations prior to each bot launch. Identify recurring failure patterns, group issues by type, quantify impact, and block launches when failure rates or severity thresholds are too high.

• Run regression testing: Re-test after model, prompt, workflow, or product updates to detect regressions introduced by changes. Escalate issues before they reach customers.

• Monitor and correct rule drift: Review dashboards weekly to spot drift or unexpected rule behavior. Use MCP/Cursor to recalibrate rules, and partner with Customer Success to prioritize fixes based on customer impact.

• Support quality reporting: Collaborate with the Business Analyst to define metrics and requirements for pre-launch quality reports, including failure modes, severity summaries, readiness status, and recommended next steps.

Qualifications

• 3+ years of experience in software QA, test case development, conversation review, trust & safety, or a closely related role

• Exceptional attention to detail and consistency—you catch edge cases others miss

• Strong written communication skills with the ability to clearly explain what’s broken and why it matters

• Strong pattern recognition skills—you identify themes across large volumes of conversations, not just individual failures

• Comfort using modern AI tools (e.g., Cursor) in day-to-day workflows

• Proactive ownership and risk awareness—you flag issues early rather than waiting for deadlines

Nice to Have

• Experience writing guidelines, rubrics, rules, prompts, or evaluation criteria that define pass/fail outcomes

• Background reviewing voice or chat interactions at scale

• Adversarial testing experience (e.g., fraud detection, cybersecurity, abuse testing, or edge-case analysis)

• Healthcare or medical practice experience

• Startup experience building QA or quality processes from scratch

What Success Looks Like

• Bots go live without customers discovering preventable issues

• Evaluation rules catch 80%+ of failure modes during simulation rather than after launch

• Regressions introduced by bot updates are identified and resolved before production

• You can clearly explain failure patterns in any bot you’ve reviewed without needing to look them up

• Rules remain calibrated—drift is detected and corrected weekly before compounding

• Pre-launch quality reports clearly communicate what is ready, what is not, and why

About Hello Patient

Hello Patient is transforming healthcare communication through advanced conversational AI. After spending 3.5 years scaling Carbon Health's technology during its hypergrowth, our founding team witnessed firsthand how much clinical time gets wasted on basic communication—great clinicians buried in administrative work, patients frustrated by missed connections and phone tag.

Our enterprise-grade AI platform proactively engages patients through perfectly timed calls and texts, naturally resolving complex interactions end-to-end—from patient acquisition and scheduling to retention. Built with security and compliance at its core, our technology integrates seamlessly with existing healthcare infrastructure to deliver 24/7 patient engagement while enabling staff to focus on what matters most: providing exceptional care.

Backed by Scale and 8VC, we're building the future of healthcare communication, one conversation at a time.