Study: What If You Showed Recruiters the Same Resume Twice?

Kickresume's goal is to help people improve their resumes and land their dream jobs. To do that effectively, you first need to understand your target audience. In this case, that means taking a good look at how recruiters work.

We wanted to see if recruiters are consistent in their choices, both with each other and with themselves. 

• Will multiple recruiters agree if a candidate is a good fit for a job? 
• Will the same recruiter be consistent with their choice when given the same resume twice? 
• And, is there a potential for automation?

We asked a surveyed sample of recruiters to look at over 12,000 pairs of candidate's resumes and job ads. We're sharing what we learned with you, whether you're a job seeker or just curious about how hiring works.

TLDR: What did we discover?

If you're not a fan of long(ish) reports, we've got you covered. 

We've distilled our research into three answers to three essential questions.

Questions we asked:

1. How consistent are multiple recruiters in assessing the same candidates?
2. Are recruiters consistent when assessing the same resume twice?
3. How does AI compare to human recruiters in assessing resumes?

Key findings:

1. Recruiters are subjective. Recruiters often disagreed among themselves about whether candidates were suited for a job. On average, the level of agreement between the recruiters was roughly half-way between random and perfect agreement. 
2. Recruiters are inconsistent (and a bit unpredictable). When a recruiter was presented with the same resume twice, they could consistently identify unsuitable candidates. However, when a candidate was deemed suitable, recruiters were consistent only 40% of the time, with their decisions appearing more random in 60% of cases.
3. Recruiters perform similarly to AI. AI demonstrated a similar level of consistency in assessing resumes compared to human recruiters.

In the following chapters, we'll offer a deeper dive into the data and methodology that led us to these discoveries.

How exactly did we examine the recruitment process?

Is there such a thing as a good candidate? 

Currently, there is no tool that can reliably determine if a candidate is "good," nor is there a metric available to measure how good the candidate is.

That’s what recruiters are for. They're the ones who decide whether you’re a good candidate. They decide whether you get an interview or not. 

But are they good at what they do? Would multiple recruiters agree on whether a candidate is a good match for a certain job? And what if you show a recruiter the same resume twice? 

In theory, if you’re a good candidate, recruiters should agree on that. It’s their job to look through your resume, your experiences, and skills to assess how good of a fit you are. 

Based on this premise, we asked multiple recruiters to look at over 12,000 pairs of candidate's resumes and job ads. They were tasked with assessing whether the candidate is a good match for the given job. 

They answered either:

• Yes (if the candidate was a good match) or
• No (if the candidate wasn’t a good match)

#1 Finding: Recruiters showed a low level of consistency among themselves

For the first part of our study, each recruiter was given the same set of 100 pairs. A pair, in this context, consists of a candidate's resume and a job ad. This allowed us to assess how consistent the recruiters were with each other in their assessments.

To put a numerical value on this consistency we used a metric called Cohen's Kappa. 

Cohen's Kappa is a measure of agreement between two dependent categorical samples, and you use it whenever you want to know if two raters' measurements are in agreement. (Source: Datatab)

When Cohen's Kappa score 0, it's similar to random chance. When it's 1, it signifies perfect agreement.

In academic literature, a Kappa score of 0.8 is typically considered indicative of strong agreement.

To illustrate the consistency levels among surveyed sample of recruiters, we've compiled a table with Cohen's Kappa scores, that offers insight into the level of consistency among themselves:

Our results show that the average Cohen's Kappa score was 0.49, which falls roughly half-way between random chance and perfect agreement.

This result indicates that the level of agreement among recruiters was quite low. In essence, the recruiters didn't consistently agree with each other when assessing whether candidates were suitable for specific job positions.

One might expect that recruiters, who specialize in candidate evaluation, would generally agree on whether a candidate is a good fit for a job. However, our findings indicate otherwise. Even when presented with the exact same pairs of candidate resumes and job ads, their decisions diferred. 

This might not be encouraging news for job seekers who expect a higher level of reliability in the recruitment process.

#2 Finding: What happens when you show a recruiter the same resume twice?

In the second part of our study, we presented recruiters with the same pair of resume and job ad they had previously assessed. 

Without them being aware of it.

What you'd expect is that the recruiter's choice would remain the same, meaning they would consistently say "yes" again if the candidate was a good match before, and "no" if the candidate wasn't a good match. 

To some extent, this expectation held true. Recruiters consistently identified candidate's resumes which weren't suitable for the position.

However, when it came to good resumes (= good candidates), the results took an unexpected turn.

Our findings suggest that when a candidate is deemed as suitable for a position, he was consistently identified as suitable only 40% of the time (i.e., the recruiter said 'yes' on both occasions), while in 60% of cases, recruiters' decisions appeared to be more random.

In other words, if you're a strong candidate, your chances of securing an interview are around 70%. Or conversely, you have a 30% chance of not being selected for an interview, despite being a good candidate.

Even if you hand in your good application to the same recruiter. 

These results shed light on just how unpredictable recruiters' assessments are. 

So, if you've ever sent out your resume, confident that you're a perfect fit for a job, only to hear nothing back — perhaps, it isn't your fault after all.

#3 Finding: AI is as good as human recruiters

Currently, there are several AI candidate matching tools on the market. While the well-known ATS software relies on filtering resumes based on keywords, the more advanced AI matching tools promise to understands candidates as humans with career profiles. 

This raises important questions: How effective are these advanced AI matching tools in practice? Can they reliably substitute a person when it comes to matching the best candidate to a job? 

To this end, we used an industry-leading AI job matching tool to assess the same resume and job ad pairs we gave to the recruiters to seek an answer to these questions. 

We again looked at Cohen's Kappa, the similarity metric we mentioned before, to put a number to the degree of agreement between an AI and a human recruiter. 

What we found is that the average agreement between an AI matchmaker and our set of human recruiters had a Cohen's Kappa score of 0.45.

This isn't far off from the average level of agreement between human recruiters. 

This suggests that the AI matching tool performs similarly well to human recruiters. 

So, to come back to the question we proposed at the beginning, whether there's any potential for automation in recruitment — it seems there is. 

Conclusion

The recruitment process is undergoing significant changes. With a growing pool of job seekers, there's a pressing need to effectively evaluate candidates and their qualifications. 

Our study delved into the world of recruitment, examining how recruiters assess candidates and whether AI could be a game-changer in the hiring process. 

Here's what we discovered:

1. Recruiters are subjective. Recruiters don't consistently agree on candidates' suitability for job positions, even when assessing the same candidates. 
2. Recruiters are inconsistent (and a bit unpredictable). When recruiters were shown the same resume twice (without being aware of it), their decisions were inconsistent, particularly when assessing suitable candidates. 
3. Recruiters perform similarly to AI. AI demonstrated a level of consistency similar to human recruiters, suggesting the potential for automation in recruitment.

The hiring process appears to be far from standardized. Quite the opposite, it often leans towards unpredictability. And unpredictability is the last thing you'd want from a recruiter who has your job prospects in their hands. 

For those on the job hunt, these findings may seem somewhat discouraging. However, there are some actionable takes you can take from this report.

Here are things you can do as a job seeker:

One approach to navigate this uncertainty is to cast a wider net. Instead of spending excessive time tailoring each application to perfection, consider applying to a larger number of jobs. 

This doesn't mean ignoring the quality of your resume completely — as we've seen, recruiters are quite consistent in identifying poor resumes. But rather than striving for excellence, aim for a really good resume. This approach allows you to have more time to craft multiple applications.

Increasing the number of applications may enhance your chances of securing interviews, given the unpredictable nature of recruiters' assessments. 

In a competitive job market, increasing your opportunities can be a strategic move to improve your chances of landing that dream job.

And remember, once you do get that interview, you still haven't won the battle. Prepare for your job interview with the AI Job Interview Questions Generator and be one step ahead.

About Kickresume

Kickresume is a popular AI career hub that offers a wide range of AI tools and resources for both job seekers and employers. With over 3,500,000 users worldwide, Kickresume is committed to helping people land their dream jobs by providing innovative and user-friendly solutions and also sharing their expertise on job search strategies.