The Good Test

Modern-day employees take a lot of assessments (i.e. surveys, tests, questionnaires, etc.) on a regular basis. Sometimes, it can be an overwhelming amount. They can be personality and ability tests that are used for selection or development purposes; they can be attitude surveys to monitor how employees feel about the job and the organization; they can be engagement surveys used to assess the level of connection and motivation one has with the organization; the list goes on.

Because of the increasing needs of employee assessments, thousands of tests and surveys developed by different vendors are being used in organizations across the world. However, some of the instruments that are mandatory by some organizations may not yield expected results and can even be harmful to both the individuals and the units.

There are two basic standards to tell if an assessment instrument is good to use: reliability and validity.

Reliability refers to the consistency of assessment outcomes. There is internal consistency that is used to indicate whether or not all the items in an instrument are measuring the same concept. For example, if you are taking an employee engagement survey that asks you questions about your satisfaction, commitment, intention to leave the organization and more, this instrument probably has very low internal consistency. It’s due to the fact that some of these questions are just all over the place and simply not about engagement.

There is also test-retest reliability to indicate whether or not the test result holds true from one administration to another. To put it simply, if a personality test tells you that you are an introvert when you took it the first time, but shows that you are an outgoing person when you re-take it a year later, this test may have some serious reliability issues. Such reliability issues have been widely found in one of the most popular personality inventories—the Myers-Briggs Type Indicator (MBTI). Some studies have shown that even though people take the test again within a short time period, 50 percent of people’s MBTI types changed on a second take.[1]

The validity of an assessment refers to whether or not it measures what it is supposed to measure. The validity indicator that is the easiest to examine is the face validity, which means you, as a test taker, can somewhat tell what the items are measuring by reading them. If you find a lot of intention-to-leave and job satisfaction items in your engagement survey, this survey probably already did a poor job in measuring employee engagement.

Content and construct validities are also very important to an assessment—they represent whether the ideas or the concepts that the assessment claims to measure have really be measured. For example, many researchers have failed to replicate the personality types theorized by MBTI, while others raise questions on whether there are meaningful differences between some of those 16 personality types in that particular assessment.^[2]

Criteria validity is another important validity indicator that shows the relationships between the assessment and important work/life outcomes. A leadership assessment that can truly reflect one’s leadership ability will probably be highly correlated with one’s leadership effectiveness score. The result of a valid employee engagement assessment should correlate well with one’s performance and work motivation.

A bad test that is neither reliable nor valid can be misleading when the test results will be used for making important decisions. It can result in selecting the wrong person for the position, diving into a profession that does not fit one’s true characteristics or missing important developmental opportunities. If decision-makers follow the guidance from the results of a bad assessment, it can make the efforts made toward improvement become a waste of time and resources — and will eventually harm individual and organizational performance and well-being.

All the reliability and validity indicators mentioned above can be calculated through certain statistical procedures by data scientists. To tell the quality of a test, it is important to let professionals examine the test to see if the important reliability and validity indicators are up to a certain standard, including the research evidence on which the test are based.

Don’t waste time and money on the bad tests anymore.

Note: Myers-Briggs Type Indicator and MBTI are trademarks of the MBTI Trust, Inc.

[1] Howes, R.J., & Carskadon, T.G. (1979). Test-retest reliabilities of the Myers-Briggs Type Indicator as a function of mood changes. Research in Psychological Type, 2, 67-72.

[2] Pittenger, D.J. (1993). Measuring the MBTI…And coming up short. Journal of Career Planning and Employment, 54, 48-53.