Your new roleThe Education & Research portfolio is seeking an experienced Senior AI Test Analyst to drive the quality, safety, and reliability evaluation of our enterprise Large Language Model (LLM) centric solutions. You will be responsible for ensuring their probabilistic outputs align with our business standards, user requirements, accessibility obligations and risk governance, and for producing the formal test artefacts required by client's Digital Project Lifecycle.
Key Responsibilities
- Evaluation Framework & Dataset Design: Build and maintain comprehensive golden evaluation datasets, benchmark suites, and testing rubrics tailored to our core business use cases.
- Hallucination & Drift Monitoring: Systematically identify and quantify hallucinations, toxic outputs, and model drift when the vendor updates underlying model versions.
- Automated Evaluation Pipelines: Implement and manage LLM-as-a-judge frameworks or evaluation tools (e.g., DeepEval, Promptfoo, Langfuse etc) to scale test execution
- Vendor & Root Cause Triage: Analyse performance gaps and trace whether failures stem from prompt logic, application retrieval layers (RAG), or the underlying SaaS model API.
- Integration & Non-functional Testing: Test integration points including the Canvas learning management system (LTI) and single sign-on, and validate non-functional behaviour covering latency, throughput, capacity limits, failover, role based access and cost per interaction.
- Accessibility & Inclusive Testing: Validate solutions against WCAG 2.2 AA and QUT accessibility requirements, coordinating with accessibility specialists where independent review is required.
Required:
- Demonstrated Experience: 7+ yrs in software quality assurance or test engineering, including ownership of test strategy and planning artefacts on enterprise or integration heavy projects, with a minimum of 2 yrs evaluating AI, LLM or machine learning based systems in a production or pre-production setting using AI Eval methodologies, goldens and benchmarks. Adjacent evidence will be considered, including model validation, search relevance tuning, content safety assurance or data quality assurance in non-deterministic environments.
- Technical Skills: Proficiency in Python for test automation and data manipulation, and in REST API testing. Comfortable working with logs, traces and structured evaluation output. Hands-on familiarity with an LLM evaluation framework is desirable rather than essential.
- AI Competency: Practical understanding of LLM behaviour, token limits, temperature and sampling settings, embeddings, retrieval, context window constraints, and how vendor model updates change system behaviour.
- ISTQB Certified Tester - AI (CT-AI) or Agentic AI equivalent.
- LLM red teaming experience and familiarity with OWASP frameworks
- You are joining a dynamic team committed to shaping and elevating Queensland's future to meet the evolving needs of our industries, ensuring prosperity and agility in our economic landscape.
- We connect individuals with quality training and skills development, opening doors to new opportunities and economic participation.
- A career with the department means making a real difference - join us in building a skilled, adaptable, and vibrant workforce for Queensland.
#3010904
About us
At Hays, we invest in lifelong partnerships that empower people and businesses to succeed. You are at the heart of everything we do. With over 50 years’ experience and a workforce of more than 10,000 people across 32 countries, we collectively put our customers, at the heart of everything we do.
Drawing on knowledge that’s unique to our scale, our commitment to understanding your needs and our ability to meaningfully innovate we are more than just a specialist recruitment business. By providing advice, insights and expertise on issues you face in today’s fast paced world of work, we’ll help you make the right decisions for tomorrow.
Looking for the right role for you?
At Hays we know that finding the right role at the right time takes careful consideration. As your lifelong career partners, we’re by your side, every step of the way.
Whether you’re seeking a new opportunity or support in navigating a career change, we bring the expertise and the connections to partner with you to help you realise your ambition.
Providing an unmatched breadth of jobs across industries and professions whilst leveraging our knowledge, employer relationships, learning opportunities and support, we can offer you roles that challenge and excite you.
Take the next step at hays.com.au
