AI Test Engineer:

The role in one line: A hands-on engineer who proves our AI features are accurate, safe and stable before and after every change, and who can build that testing setup alone.
What this person will actually do
    • Build automated accuracy checks for the AI
    • Make those checks a gate in the CI pipeline,]
    • Build and maintain test datasets
    • Test the whole pipeline end to end:
    • Watch quality in production using LLM tracing tools (Arize Phoenix, LangSmith) to spot drift, rising failures and cost or latency jumps.
    • Run safety and robustness testing: prompt injection inside documents, hallucinated fields, PHI leaking into logs, and inconsistent answers to the same input.
    • Produce evidence for a regulated product. Test reports need to stand up in a medical-device-style audit (IEC 62304, HIPAA).
Must-have skills (screen hard on these)
    1. Strong Python test automation: pytest, API testing, CI pipelines (GitLab).
    2. LLM evaluation hands-on: has built eval sets and scoring, not just “used ChatGPT.” Ask them to name the metrics they used and how they set pass/fail thresholds.
    3. Experience with at least one eval or tracing tool: LangSmith, Arize Phoenix, DeepEval, Ragas, promptfoo, or similar.
    4. Understands non-deterministic testing: how to test something that gives slightly different answers each run (repeat runs, tolerances, LLM-as-judge and its limits).
    5. Document or vision AI exposure: OCR or extraction from PDFs and images is a big plus for us.
    6. Solo-capable: has set up a test approach from scratch, not only executed someone else’s.
Good-to-have
    • LangChain / LangGraph. Useful, but not essential: our AI Doc calls Gemini directly through Google’s SDK. Don’t reject strong candidates for lacking LangGraph.
    • Google Cloud / Vertex AI, Django APIs, Playwright.
    • Healthcare or regulated-domain testing, and handling of PHI and de-identified data.
    • Performance and cost testing of LLM calls (latency, tokens).