Utility and a Decision Theory Approach to Selection Learning Objectives: After you have completed this exercise, you will: 1. Identify the differences between the classical validity approach and the decision theory app

Utility and a Decision Theory Approach to Selection
Learning Objectives: After you have completed this exercise, you will:
1. Identify the differences between the classical validity approach and the
decision theory approach to selection.
2. Define the concepts of base rate, selection ratio, validity, and utility.
3. Identify how the utility of a selection procedure is determined by base
rate, selection ratio, validity and cost.
4. Calculate the utility of a proposed selection procedure.
5. Determine how the utility of a selection device changes as the base rate
and selection ratio change.
As opposed to the classical validity approach to selection, which
concentrates only on obtaining the highest possible multiple correlation (R)
between the selection battery scores and the criterion (e.g., a measure of job
performance), the decision theory approach to selection concentrates on
the consequences and outcomes of prediction. An answer is sought to the
question “How much better would we do with the new selection process?”
This is really a question concerning the utility or practical usefulness of the
new procedure. Utility determination involves comparing outcomes
obtained with and without the new procedure. Utility is a function of the
base rate, the selection ratio, the validity of the procedure being evaluated,
and cost.
The base rate is defined as the proportion of present employees who are
considered to be successful on their jobs and is an indication of how well
the present selection system is functioning. The higher the base rate, the
lower the potential utility of the new selection process, since the present
selection process is doing a relatively good job of hiring successful job
performers, leaving little opportunity for improvement. For example, with a
base rate of 95%, there is not much room for improvement, and even a very
valid test would not enable much better selection. The lower the base rate,
the greater the potential utility of the same new selection process, since there
is more opportunity for improvement. For example, a base rate of 50%
would indicate much room for improvement, and the same selection
procedure would have much more utility.
The selection ratio is defined by a fraction. In the numerator is the number
of people to be hired (or equivalently, the number of job openings). In the
denominator is the number of applicants possessing at least the minimum
requirements for the job so that they may realistically be considered for the
job. The higher the selection ratio, the lower the potential utility of the new
process – if the organization must hire a large percentage of applicants, the
less selective it may be. If the selection ratio equals 1.0, even a very valid
test has no utility, because every applicant must be hired. The lower the
selection ratio, the higher the potential utility of the new process, since the
organization may be more selective in whom it hires. With a low selection
ratio, only the very best applicants need be hired, and even predictors with
relatively low validities can have good utility.
Validity is the degree of relationship between the new selection process and
the criterion. The greater this relationship, the greater the potential utility of
the new process.
Cost refers to such things as the cost of administering the new process,
relative cost to the organization of incorrectly versus correctly selecting an
applicant, and incorrectly versus correctly rejecting an applicant.
For each of the questions below, show ALL your calculations.
Exercise:
Part 1: The following diagram represents the relationship between
employees’ test scores and their job performance scores for the position
Management Trainee. All persons represented in this diagram were hired
using a structured interview; i.e., test scores were not used to hire these
people. Job performance scores were obtained from their managers at the
first review period (approximately six months after hire). The summary
performance evaluation was on a five-point scale with 5 being “excellent,” 4
being “good,” 3 being “average,” 2 being “fair,” and 1 being “poor.”
Trainees with performance ratings of “average” to “excellent” were
classified as satisfactory; those with ratings of “fair” and “poor” were
classified as unsatisfactory. The numbers in the diagram represent the
proportions of trainees in each quadrant of the diagram.
Satisfactory
Job Performance
Scores
Unsatisfactory
Reject Hire
Test Scores
Figure 20-1: Relationship between test scores
and job performance scores for
Management Trainees in the ABC Company
1. Define the term “base rate,” and determine its value in this situation.
2. Define the term “selection ratio,” and calculate its value here if we used
the test to hire new applicants. (Assume that everyone who was hired
using the structured interview possesses at least the minimum
qualifications for the job.)
3. If the test were used in addition to the existing interview to select
trainees, what would be the percent of satisfactory employees selected
( = the success ratio = the new base rate)?
4. What is the approximate utility of the new selection device (i.e., the
improvement due to the test)?
5. Would you recommend that this test be added to the selection procedure
for hiring trainees? Why or why not?
.07
.15
.10
.68
Part 2: The following diagram represents the same situation as before,
except the test cutoff score has been changed to reflect a more favorable
selection ratio for this organization.
Satisfactory
Job Performance
Scores
Unsatisfactory
Reject Hire
Test Scores
Figure 20-2: Changing the test cutoff
score to reflect a more favorable
selection ratio
6. Determine the base rate, selection ratio, percent of satisfactory employees
using the test, and the utility of the test, as before. Compare your results
with the previous problem. How does making the selection ratio more
favorable to the organization (i.e., lower) change the utility of the same
selection device? Why?
7. Why is the selection ratio in this situation considered more favorable to
the organization than the selection ratio in the previous problem?
8. If a selection ratio is more favorable to the organization, is it more or less
favorable from the standpoint of applicants for the job? Why?
9. How may an organization make the selection ratio more favorable? Are
there costs involved in doing this? What should an organization consider
in determining whether or not to decrease the selection ratio?
.12
.10
.05
.73
Part 3: The following diagram represents the same situation as in Part 1,
except the base rate is higher. This might be the result if, for example, the
organization is willing to accept a lower level of performance as satisfactory.
Satisfactory
Job Performance
Scores
Unsatisfactory
Reject Hire
Test Scores
Figure 20-3: Using a higher base rate
10. Determine the base rate, selection ratio, percent of satisfactory
employees using the test, and the utility of the test, as before.
11. Compare your results with Part 1. How does having a higher base rate
change the utility of the same selection device? Why?
.17
.22
.03
.58