automation tester Work, Jobs in leeds (with Salaries)

AI Tester
SmartSearch —Ilkley
- Full-time
- Hybrid work
- Master's degree
- SQL
- Analysis skills
- Employee assistance programme
Quick apply
Software Tester
Benenden Health —York
- £47,500 a year
- Hybrid work
- Agile
- Employee assistance programme
- Referral programme
12d
Automation Engineer
interactive investor —Leeds
- From £40,000 a year
- Full-time
- Hybrid work
- Azure
- Microsoft Office
- AWS
- Annual leave
- Employee discount
Quick apply
Senior Automation Tester (Python) – Agentic AI & LLM Systems (Hybrid)
Resillion —Leeds
- Remote
- Azure
- Master's degree
- AI
Quick apply
System Analyst
Hiscox —York
- Full-time
- Azure
- SQL
- APIs
12d
Release & Environments Engineer
Pharmacy2U —Leeds
- Full-time
- Hybrid work
- Azure
- Communication skills
- Python
- Annual leave
- Employee assistance programme
Quick apply
Senior Delivery Manager
Informed Solutions —Leeds
- Full-time
- Hybrid work
- Azure
- Continuous improvement
- Contracts
- Gym membership
- Referral programme
Quick apply
Lead DevSecOps Engineer
TransUnion —Leeds
- Full-time
- Mentoring
- Computer networking
- APIs
- Employee stock purchase plan
- Annual leave

Discover related jobs

Based on your search for automation tester jobs.

Automation Engineer
interactive investor —Leeds
Work closely with all areas of the business to complete testing on behalf of internal and external customers. Understanding of financial products and services,…
From £40,000 a year
Quick apply
Release & Environments Engineer
Pharmacy2U —Leeds
Flexible hybrid working to help balance work and home life. Regular social events throughout the year. Support sprint and release cycles by ensuring lower…
Quick apply

I want to receive the latest job alerts for automation tester in leeds

By signing in to your account, you agree to SimplyHired's Terms of Service and consent to our Cookie and Privacy Policy.

AI Tester

SmartSearch -
Ilkley

Quick apply

Job details

Permanent | Full-time

Benefits

Cycle to work scheme
Company pension
Employee assistance programme
Private medical insurance
Paid volunteer time

Qualifications

Law
Test automation
Master's degree
SQL
Analysis skills
Scripting
APIs
Software testing
Test cases
AI
Python

Full job description

SmartSearch's distinctive Anti-Money Laundering verification software protects our clients by offering the most advanced and comprehensive features available from an AML provider.

SmartSearch has grown rapidly by fostering an incredibly collaborative and supportive culture. As we continue our ambitious growth plans, we will strive to remain a truly exciting, rewarding, and unique place to work.

HOW WILL YOU MAKE A DIFFERENCE?

We're looking for an AI Quality Analysist to help validate and improve Generative AI products and features. This role focuses on testing LLM-based systems, prompt workflows, agent behaviours, RAG pipelines, and AI-driven user experiences. You'll work closely with product, engineering, and AI teams to identify quality risks, design robust test strategies, and ensure our GenAI solutions are reliable, safe, and production ready.

VARIED DAY TO DAY RESPONSIBILITIES

Design and execute test plans for Generative AI applications, including chat, summarization, extraction, classification, and agent workflows
Validate LLM outputs for quality, relevance, accuracy, consistency, tone, and policy compliance
Test prompt behaviour across normal, edge, and adversarial scenarios
Evaluate RAG systems for retrieval quality, grounding, hallucination risk, and citation accuracy
Create test cases for multi-step AI workflows, tool use, memory, and orchestration logic
Partner with engineers and product managers to define acceptance criteria for AI features
Identify and document defects, regressions, safety issues, and model failure patterns
Build and maintain test datasets, evaluation rubrics, and benchmark suites
Support automated testing for AI features, APIs, and end-to-end workflows
Track model and prompt changes to assess quality impact over time
Contribute to release readiness by reporting on AI quality metrics, risks, and recommendations

WHAT ARE WE LOOKING FOR IN A CANDIDATE?

3+ years in QA, software testing, or test engineering
Hands-on experience testing AI/ML or Generative AI applications
Strong understanding of LLM behaviour, including hallucinations, prompt sensitivity, grounding, and non-deterministic outputs
Experience writing detailed test cases, bug reports, and test summaries
Familiarity with API testing tools and test automation frameworks
Ability to evaluate model outputs using structured rubrics and human judgment
Strong analytical skills and attention to detail
Comfortable working cross-functionally in a fast-moving product environment

Preferred Qualifications

Experience testing chatbots, copilots, agents, or RAG-based applications
Familiarity with prompt engineering and prompt/version evaluation
Experience with Python, SQL, or scripting for test data and automation
Knowledge of AI safety, red teaming, bias testing, and policy evaluation
Experience with model evaluation frameworks, telemetry, or experimentation
Exposure to cloud AI platforms and LLM providers

WHAT IS LIFE LIKE AT SMARTSEARCH?

We are a multi-award winning Tech company with an aspirational mentality
Some of our most recent recognitions include: named in the renowned RegTech100 list for 2024, listed in the Top 100 Fasted Growing Tech Companies by Northern Tech Awards 2024 as well as being named Technology Provider of the Year by Corporate Finance Awards 2024
We have been Great Place To Work Certified since 2022
There are excellent progression opportunities due to our growth and you will have personal development goals, regular feedback and support
We are a diverse and inclusive team committed to promoting Diversity & Inclusion and Social Responsibility. Through our DE&I group, charitable initiatives and support for local schools, we actively foster a positive Impact on our community

COMPANY BENEFITS

Our comprehensive benefit package includes:

25 days holiday rising to 30 with each year of service
Private Medical Insurance covering dental and optical
Company pension scheme
Life Assurance - 4x your annual salary
1 day paid volunteering per year
Enhanced maternity / paternity offerings
Employee Assistance Programme
Cycle to work scheme
Access to a gym

Quick apply

Refine Your Search

automation tester jobs in leeds

AI Tester

Software Tester

Automation Engineer

Senior Automation Tester (Python) – Agentic AI & LLM Systems (Hybrid)

System Analyst

Release & Environments Engineer

Senior Delivery Manager

Lead DevSecOps Engineer

Discover related jobs

I want to receive the latest job alerts for automation tester in leeds

Explore jobs in more locations

HOW WILL YOU MAKE A DIFFERENCE?

VARIED DAY TO DAY RESPONSIBILITIES

WHAT ARE WE LOOKING FOR IN A CANDIDATE?

WHAT IS LIFE LIKE AT SMARTSEARCH?

COMPANY BENEFITS

Jobseeker tools

Employer Tools

Browse

Stay Connected