AI Test Engineer:
The role in one line: A hands-on engineer who proves our AI features are accurate, safe and stable before and after every change, and who can build that testing setup alone.
What this person will actually do
- Build automated accuracy checks for the AI
- Make those checks a gate in the CI pipeline,]
- Build and maintain test datasets
- Test the whole pipeline end to end:
- Watch quality in production using LLM tracing tools (Arize Phoenix, LangSmith) to spot drift, rising failures and cost or latency jumps.
- Run safety and robustness testing: prompt injection inside documents, hallucinated fields, PHI leaking into logs, and inconsistent answers to the same input.
- Produce evidence for a regulated product. Test reports need to stand up in a medical-device-style audit (IEC 62304, HIPAA).
Must-have skills (screen hard on these)
- Strong Python test automation: pytest, API testing, CI pipelines (GitLab).
- LLM evaluation hands-on: has built eval sets and scoring, not just “used ChatGPT.” Ask them to name the metrics they used and how they set pass/fail thresholds.
- Experience with at least one eval or tracing tool: LangSmith, Arize Phoenix, DeepEval, Ragas, promptfoo, or similar.
- Understands non-deterministic testing: how to test something that gives slightly different answers each run (repeat runs, tolerances, LLM-as-judge and its limits).
- Document or vision AI exposure: OCR or extraction from PDFs and images is a big plus for us.
- Solo-capable: has set up a test approach from scratch, not only executed someone else’s.
Good-to-have
- LangChain / LangGraph. Useful, but not essential: our AI Doc calls Gemini directly through Google’s SDK. Don’t reject strong candidates for lacking LangGraph.
- Google Cloud / Vertex AI, Django APIs, Playwright.
- Healthcare or regulated-domain testing, and handling of PHI and de-identified data.
- Performance and cost testing of LLM calls (latency, tokens).