# Introduction
We ran this A/B test to see whether different, more personalised subject lines would encourage more jobseekers to click through to vacancies from their job alert emails. We tested five variants against the current subject line for job alert emails, making six variants in total. We also measured how relevant the job listings were to jobseekers.
# Hypothesis
IF job alert emails have a more personalised subject line THEN more jobseekers will open their job alert email and click through to the vacancies within.
# Variants
- Present version: ‘Your job alert from Teaching Vacancies’
- Variant 1: [Job title] at [School] and [number of new jobs minus 1] other new jobs
- eg Maths Teacher at Lambeth Academy and 4 other new jobs
- Variant 2: [School name] is hiring for a [job title] - and [number of new jobs minus 1] other new jobs
- eg Lambeth Academy is hiring for a Maths Teacher - and 4 other new jobs
- Variant 3: Your job alert for [keywords]
- eg Your job alert for Mathematics
- Variant 4: [Number of new jobs] new jobs match your criteria
- eg 6 new jobs match your criteria
- Variant 5: Your job alert: [Number of new jobs] new jobs match your criteria
- eg Your job alert: 6 new jobs match your criteria
# Who we tested with
We tested with 127,231 jobseekers between 27 January 2022 and 11 March 2022. 1/6th of users who receive job alerts were assigned each variant.
# Findings
- More personalised subject lines do increase the CTR to a vacancy from a job alert email
- all variants had a higher CTR than the present subject line, for which the CTR was 7.58%
- variants 1, 2 and 4 were significantly different to the present subject line at the 80% confidence level
- Variant 2 had the highest CTR at 8.17%
- this was significantly different to the present subject line at the 95% confidence level
- however, there was insufficient data to show that variant 2 generated a higher CTR than variants 1 and 4, even at the 80% confidence level
- A higher CTR does not always guarantee that results seem more relevant to the jobseeker
- we measured the document download rate (DDR) - the proportion of users who were sent a job alert with a particular variant who then went on to download the documents on a job listing - as an indicator of how relevant the results were to users
- with a higher CTR, we would have expected to see a higher DDR
- variant 2 had the second-highest DDR and variant 4 had the highest
- however, there were insufficient data to show a significant difference or similarity between the two, even at the 80% confidence level
- this was unexpected because the results the jobseekers saw were not changed - the only difference was the subject line of the job alert email sent to them
- one explanation for this could be that the subject line influenced the behaviour of jobseekers, and how they continued to engage with the listings
# Next steps
Based on these findings we decided to use variant 2 for all job alert emails. This is because it generated the highest CTR to a vacancy and results still appeared relevant to jobseekers, and certainly more so than the present subject line.
# Screenshots
