There are currently three ways in which a jobseeker can set up a new job alert:
- Click the ‘Create job alert’ link at the top of every page on the site. This then invites the user to create a job alert via a form from scratch, setting each of the criteria manually.
- Carry out a site search and then click the blue ‘Receive a job alert whenever a job matching this search is listed’ banner that appears at the bottom of the search results page. In this case the fields in the job alert creation form are populated automatically with the criteria they included in their site search.
- View a vacancy and then click on the ‘Get a job alert for when similar jobs are listed’ link next to the ‘Similar jobs’ section. In this case the fields in the job alert creation form are populated automatically based on the listing they viewed. For example, by default the job alert might be set up for jobs within 10 miles of the listing and for a similar education phase.
In this last case, one of the criteria that was automatically populated was the ‘working patterns’ field. For example, if a user viewed a vacancy with the working pattern ‘full time only’ then a job alert would be created which, by default, only sent alerts for new vacancies which had the option of a full time working pattern.
However, analysis of the first month’s usage of this functionality indicated that:
- 41% of job alerts created from the ‘similar jobs’ section of a listing had been created without a working pattern specified, indicating that these users had taken explicit action to remove the working pattern
- ‘Similar jobs’ job alert emails without a working pattern specified had a significantly higher CTR (13.4%) than ones with a working pattern specified (8.1%), indicating that the former were perceived as more relevant, despite being less specific.
We wanted to test whether leaving the working patterns on the new job alert blank instead of auto-populating it would improve the relevancy of job alert emails for jobseekers.
# Hypothesis
If we stop setting working patterns automatically, the overall relevance of ‘similar jobs’ alert emails will increase.
# Who we tested with
We sent 5,045 job alert emails to users between 29th September and 18th October 2021. 2,688 were sent with an automatically set working pattern - although a user could have chosen to remove or change the working pattern manually - while 2,357 were sent without an automatically set working pattern.
# Findings
If the test were to be repeated, we would expect:
- 13.7 +/- 1.3% of job alert emails to be clicked on when the working pattern was set automatically - a total of 367 +/- 35 clicks
- 16.4 +/- 1.5% of job alert emails to be clicked on when the working pattern was not set automatically - a total of 387 +/- 35 clicks
So, stopping setting the working pattern automatically:
- increases the overall clickthrough rate significantly from 13.7% to 16.4%
- substantially reduces the number of irrelevant emails by ~15%
- slightly increases the total number of relevant emails by ~5%
# Next steps
Based on these findings, we decided to stop setting the working pattern automatically.