How to Evaluate Pre-Employment Assessments

Pre-employment assessments can improve the speed and accuracy of hiring decisions.

Unfortunately, not all pre-employment assessments are equally useful. How can you tell the difference?

Here are 4 critical areas that every employer should consider before making the decision to use a pre-employment assessment: Reliability, Validity, Validation, and Legal Considerations.

Reliability

A pre-employment assessment that isn’t reliable cannot be valid. Remember that.

This doesn’t mean that reliability and validity are interchangeable—it’s still possible to reliably measure something that isn’t valid or useful. More on that in the next section.

Reliability is just the necessary first step. It’s the first thing you should consider.

Here’s a checklist of the main reliability points:

• Consistency: The goal of most pre-employment assessments is to compare applicants. You can only make these comparisons if you consistently obtain the same types of information from all applicants.
• Agreement: If assessment results are to be interpreted by a person, it should be possible for another person to reach the same conclusions, based only on seeing the assessment results.
• Retest reliability: If a pre-employment assessment is administered to the same person on more than one occasion, that person should obtain similar results each time.

Tips and warning signs:

• Beware of pre-employment assessments that claim to “customize” with unique questions for each applicant—these questions might seem interesting, but it makes valid comparisons impossible.

• Beware of pre-employment assessments that require “special training” or “certification” to interpret, or that require “evaluators” to negotiate or compromise. A reliable assessment should provide relatively unambiguous results.

• Beware of pre-employment assessments that divide people up into “types” or other categories. Most people don’t actually fall neatly into these “types” or categories. Therefore, applicants will often get different results if you retest them.

Validity

A pre-employment assessment that isn’t valid cannot add value.

This doesn’t mean that validity ensures value added—it’s still possible to have a valid assessment measuring something that isn’t valuable to you. More on that in the next section.

Here’s a checklist of the main validity points:

• Face validity: Pre-employment assessments should appear to be accurate, relevant, and fair.
• Content validity: Assessments should collect relevant and useful information from applicants (and minimize collecting irrelevant or useless information).
• Predictive validity: Assessment results should statistically predict relevant outcomes. This is one of the most important considerations.

Tips and warning signs:

• Beware of pre-employment assessments that even appear to be inaccurate, irrelevant, or unfair. Applicant experience is important.

• Beware of pre-employment assessments that cannot directly provide at least some feedback and results to applicants. (Either they are running on outdated technology, or they are not scored using a reliable system.)

• Beware of pre-employment assessments that are based on a theory. If you are considering a theory-based assessment, look for published, peer-reviewed scientific evidence supporting the theory. You will probably be disappointed.

• Beware of pre-employment assessments that claim to “clone” your best employees based on similarity. Similarity is not the same as valid statistical prediction, and it does not produce the same outcomes. Be especially wary of assessments that claim to “clone your top performers”—see the point above about consistency in the reliability section.

• Beware of pre-employment assessments that use previous “success stories” as evidence of validity. The most valid assessments are customized to generate statistical predictions that meet the needs of specific job roles within specific organizations or industries.

Validation

A pre-employment assessment that is reliable and valid can add tremendous value.

But even if you know that it can work, the real question is if it’s working for you.

Here’s a checklist of the main value-added points:

• Scientific validation: Before you begin, ask for published, peer-reviewed scientific evidence demonstrating reliability and validity of the type of pre-employment assessment you’re considering.
• Statistical validation: Collect your own data over time. Analyze the results. Does the assessment actually provide you with valid statistical predictions for important outcomes?
• Hiring metrics: How does the pre-employment assessment influence applicant experience, hiring speed, cost-per-hire, etc.?

Tips and warning signs:

• Beware of assessment providers that cannot produce published, peer-reviewed scientific evidence demonstrating reliability and validity for the type of pre-employment assessment they offer.

• Beware of assessment providers that claim to have “extremely accurate” assessments, especially if they give you a very high number. Even the best pre-employment assessments can only explain part of the overall variation in relevant outcomes when they are actually tested by competent experts.

• Beware of assessment providers that claim to have special or secret knowledge that scientists aren’t aware of. They don’t.

• Beware of assessment providers that will not allow you to collect and analyze detailed information about the assessment results for yourself. If your organization doesn’t have the capability to analyze and evaluate hiring and assessment data, hire a consultant to do it. Don’t just assume that it’s working!

• Beware of over-interpreting changes in your organization, for better or worse. These changes may be due to a pre-employment assessment, but could also reflect broader trends that may change over time.

Legal Considerations

Before considering any pre-employment assessment, it’s important to understand a few legal issues.

Here’s a checklist of the main issues to consider:

• Illegal questions: Assessments cannot legally collect information about race, color, religion, sex, gender, national origin, age, disability, or genetic background.
• Disparate impact: Even if you don’t discriminate on purpose, it’s still possible for an assessment to negatively impact some groups more than others. If this happens, you need to know about it so you can take appropriate actions.
• Reasonable accommodations: Applicants with disabilities can request reasonable accommodations, and employers are required to provide them.

Tips and warning signs:

• Avoid assessments that collect information about race, color, religion, sex, gender, national origin, age, disability, or genetic background. Try to avoid assessments that might indirectly indicate such things as well. For example, some psychological assessments might reveal potential mental disorders (mental disorders are legally protected as a disability).

• Collect your own data over time. Analyze the results. Beware of assessment providers that will not allow you to collect and analyze detailed information about the assessment results for yourself. If your organization doesn’t have the capability to analyze and evaluate hiring and assessment data, hire a consultant to do it. Don’t just assume that any assessment is treating people fairly!

• Ensure that you will be capable of providing reasonable accommodations for applicants to complete the assessment process.