Schmidt SIOP 2015 Living History Interview
1
SIOP 2015 Living History Series:
An Interview With Frank L. Schmidt
Interview Questions and Answers
Jeffrey M. Cucina (Chair and SIOP Historian)
U.S. Customs and Border Protection
&
Frank L. Schmidt (Interviewee)
Tippie College of Business, University of Iowa
Gallup Organization
Introduction
The SIOP History Committee conducted its third installment of the SIOP Living History Series at
the 2015 annual SIOP conference. The goal of this series is to interview individuals who have made
historic contributions to research or practice in I-O psychology. There is perhaps no better way to learn
about the early activities of SIOP than by listening to our early contributors’ experiences. This year’s
interviewee was Dr. Frank L. Schmidt. In the personnel selection literature, there is perhaps no greater
luminary than Dr. Schmidt and his co-author Dr. John E. Hunter. This document provides Dr. Schmidt’s
narrative responses to the Living History Series interview.
Brief Overview of Dr. Schmidt
•
•
•
•
•
•
Professor Emeritus, Tippie College of Business, University of Iowa, and senior scientist with the
Gallup Organization.
Authored over 200 publications. Cited 36,213 times per Google Scholar. Current H Index is 77.
Identified by Podsakoff et al. (2008) as being in the 99th percentile for citations within
management.
Popular personnel selection textbooks cite the Schmidt and Hunter pair more frequently than
any other set of authors.
o Meta-analysis textbook (Schmidt & Hunter, 2014) and Psychological Bulletin article on
98 years of personnel selection research (Schmidt & Hunter, 1998) have been cited
4,702 and 2,792 times, respectively.
o As one research psychologist recently opined, “if you need a cite for something in
selection, it’s likely Schmidt and Hunter.”
Career has spanned both academia and practice (with positions at the U.S. Civil Service
Commission, Michigan State University, George Washington University, and University of Iowa).
Winner of numerous awards and honors
o APA Distinguished Scientific Contributions Award, 1994 (joint with J. Hunter) (Add this
one)
o SIOP’s 1995 Distinguished Scientific Contributions Award (joint with J. Hunter),
o SIOP’s 1974 Cattell Research Design Award (joint with J. Hunter),
o APS’s 2007-2008 James McKeen Cattell Fellow Award,
o American Psychological Foundation’s 2013 Gold Medal Lifetime Achievement Award for
Applications of Psychology)
Schmidt SIOP 2015 Living History Interview
2
Early life history
•
•
•
Can you tell us about your early life, where you grew up, where you went to school?
o Grew up in Jeffersontown, KY—Small town outside Louisville. Grew up on a farm.
Hunting, fishing, mischievous kid, Boy Scout leader, in trouble with teachers, etc. I
recently wrote a memoir that has lots of stories about my early life growing in
Jeffersontown. It covers my research career too, up to the present. It is available on
Amazon.com. (Under Books, type in: The Importance of Being Frank, by Frank L.
Schmidt.)
o PhD in Industrial Psychology, Purdue University, 1970
o MS in Industrial Psychology, Purdue University, 1968. No need to mention this.
o BA in Psychology, Bellarmine College, 1966.
What led you to become an Industrial Psychologist?
I was a biology major for 2.5 years before switching to psychology. If fact, I entered college
with 13 credit hours of advanced placement in biology. I learned about Industrial Psychology
in the Intro Psych course and immediately realized that was what I wanted to do.
What were you like as a graduate student? Who was your advisor? What was your dissertation
topic?
Thanks to the Russian Sputnick I had a lucrative NDEA graduate fellowship, so I had no need to
attach myself to a particular professor to get RA or TA funding. I was pretty independent and
did not have an actual mentor. The faculty members that had the biggest impression on me
were Hubert Brogden and Bill Owens. My official advisor was Joe Tiffin, but he was more of a
figurehead.
o “The relative efficiency of regression and simple unit predictor weights in applied
differential psychology.”
My dissertation research showed that regression was much less useful than most people
believed. It showed that in a large percentage of regression studies, simple unit weights (equal
weights) produce better prediction than regression weights. This was when I first realized that
researchers did not understand sampling error and did not realize how severe capitalization
on sampling error can be.
Career History
•
After graduating from Purdue, you went to Michigan State University. Can you tell us about your
professorship there?
By April 1970, I had two job offers—one from the U.S. Civil Service Commission in Washington DC
and one from the Central Intelligence Agency (CIA). I had known that the Civil Service Commission had
a big industrial psychology research program in personnel selection, but I was not aware that the CIA
did too, until they contacted me. But in the CIA job, getting any of your research studies published
would have been very hard; the security clearance procedure was complex and took years. I listened
when my professors told me the best way to launch your career was with an academic job at a good
research university. You could publish there and get visibility. Michigan State had an assistant
professorship opening for a quantitative industrial psychologist like me, so I applied there.
Schmidt SIOP 2015 Living History Interview
3
My interview visit to Michigan State was in mid May 1970, and campus protests against the
Vietnam War were at their peak there. The campus was a war zone. As we drove around on that big
campus, I saw broken windows, buildings scorched out and boarded up, fires burning, and student
tents and shanties spread out over the grassy areas. Nothing like this had happened at Purdue and I
was kind of taken aback. Nevertheless I accepted the job when they offered it. As I describe later, it
was on this job interview that I first met Jack Hunter.
•
You were there for four years, gaining tenure after just 3 (?) years. What is the secret for getting
tenure so fast?
At MSU, I did some research with Bill Crano, a social psychologist. Bill had gotten early tenure and
he told me I should apply for it too. He said my research record was as good as his was when he got
early tenure. The idea of early tenure had not crossed my mind, but because of Bill Crano I applied for
it and got it.
Nevertheless, I left MSU for the U.S. Civil Service Commission (later OPM), because I felt that as an
applied psychologist I needed “real world” experience.
•
At that point, you became a federal employee, working for the U.S. Civil Service Commission
(now called OPM). What type of work did you do at the Civil Service Commission?
The Civil Service Commission was charged by law with ensuring that all federal hiring was done on
a merit basis. That law was enacted in the 1880s to get rid of the corrupt spoils system that had
previously held sway. This is important, because the federal government is the largest employer in the
U.S. and in the world.
After Jimmy Carter defeated Gerald Ford in the 1976 Presidential election, Carter got Congress to
pass a bill called The Civil Service Reform Act of 1977. This law changed the name of the agency to the
U.S. Office of Personnel Management (OPM). It also weakened the merit-based hiring system, by
allowing a big increase in political appointees who did not have to be hired based on merit. According
to the Washington Post, when Carter was told that this could lead back to the spoils system, he
replied, “What’s wrong with the spoils system? We had that in Georgia when I was governor and it
worked fine.”
Government jobs were allocated based on the spoils system starting under President Andrew
Jackson and continuing up to passage of the 1881 Pendleton Act, which mandated merit-based federal
hiring. Francis Fukuyama, in his 2014 book, Political Order and Political Decay, shows how the spoils
system leads to incompetent and corrupt government officials and a dysfunctional government. Yet
this was the direction in which the Carter administration was pushing.
Our job at OPM was to make sure that hiring was done on the basis of merit, as required by law.
OPM had done research on the written hiring tests it had developed that showed they did a good job
of predicting the future job performance of job applicants and did so in the same way for Whites,
Blacks, and Hispanics. That is, Whites, Blacks, and Hispanics with the same test scores had the same
later job performance on average. This is the definition of test fairness. Similar research had been
done by others in private industry and academia, with the same results. The research literature was
full of studies like this. No studies concluded that the tests were predictively biased against
Schmidt SIOP 2015 Living History Interview
4
minorities. Use of these tests in hiring substantially improved the job performance of those who were
hired (cf. Schmidt, Hunter, Outerbridge, & Trattner, 1986).
Carter appointed a man named Scotty Campbell to head OPM. Campbell told us he personally met
with Carter every week. He told us that Carter did not like our hiring tests because Blacks scored lower
on them on average. Black votes had helped put him in the White House, and he and his sidekick
Andrew Young wanted to get rid of the tests so more Black applicants could be hired.
We presented Campbell with all the research showing these tests predicted performance well and
just as well for Blacks as Whites. He told us even if he wanted to believe these research results it
wouldn’t matter, because he had his orders from Carter to get rid of the tests. His deputy head of
OPM, Jule Sugarman, was even worse. Sugarman told us, “I know how these psychology studies work.
You can make the results turn out any way you want them to. And you are making them turn out
against our policy. You are not supporting our policy.” That was in 1979. We ignored him and kept on
doing what the law required.
In early 1980, Campbell and Sugarman called all of us up to their conference room on the 7th floor.
When we were all seated around the big conference table, Campbell said “You are still not supporting
our policies. We are not going to do anything about that right now. But after the election, we are
going to do some house cleaning around here. You might want to start looking for another job now.”
That never happened. Carter lost the 1980 election and Reagan became president. Scotty
Campbell was out of a job. He then took a job as Director of Human Resources at a big food company,
Sysco. After a few months we got phone calls from him. He wanted advice on which written tests he
should use in hiring at Sysco. We told him, “Scotty, we thought you didn’t believe in these tests.” He
said, “Well, things are different now.”
In the last 3 days of his presidency, Carter paved the way for the wholesale elimination of merit
based testing for federal hiring. Some years earlier, the government examination used for hiring
college graduates had been challenged by lawyers contending it was biased against minorities, but
this suit had gone nowhere because OPM had the evidence to prove this charge was not true. In the
last few days before Reagan was inaugurated, Carter agreed to a consent decree proposed by these
lawyers which eliminated that test and, in time, almost all written tests used in federal hiring. The
result over time was a major decrease in the intelligence and competence of new federal workers.
One of our OPM studies (Schmidt, Hunter, Outerbridge, & Trattner, 1986) showed that this reduction
in quality and quantity of job performance would cost the government—and the taxpayers—billions
of dollars every year.
We did not have this problem with Reagan but we had another problem. The people Reagan
appointed accepted the research findings and liked our hiring tests. But they had some reservations,
too. They said these tests are so good that we might wind up siphoning off the smartest people into
government jobs when these people should be “reserved” for the private sector—business and
industry. The message was: hire good people but not the very best people.
Then there was the Reagan-appointed economist who came in and read all our research studies
and reports. He pronounced them very high quality. But he said the government should not be paying
for research of this kind. Instead, he said, Exxon should use its excess profits to fund research of this
kind. It was good research but it should be done in the private sector, not by government. We realized
that he thought this research was really expensive and the government was spending a lot of money
Schmidt SIOP 2015 Living History Interview
5
on it. We then demonstrated to him that it was actually pretty cheap to do this research, and he
relented and said we could keep doing it.
Later in this interview, I will discuss how we developed validity generalization and meta-analysis
at OPM.
•
A few years later you put your foot back in academia, serving as a part-time professor at George
Washington University. Can you tell us about your time there?
My appointment at GWU was as an Adjunct Professor. Yet the Psych Department gave me an
office and allowed me to teach PhD courses in measurement, personnel selection, and meta-analysis.
What is even more unusual is that they allowed me to direct PhD dissertations—unheard of for an
adjunct professor. I directed the dissertations of Mike McDaniel, Ken Pearlman, Brian Stern, and
several others. However, because of university rules, the name on the paperwork had to be Jim
Mosel’s, a full professor there. That worked out to everyone’s satisfaction.
On a typical day, I would leave OPM around 9:00 and go to my GWU office. I’d return around
quitting time. It was while sitting in my GWU office one day that I got the idea for validity
generalization.
•
After 9.5 (This was after 11 years) years, you left GW and went to the University of Iowa. What
drew you there?
There is an unusual story behind my leaving OPM and going to the University of Iowa business
college. Duane Thompson was an I-O psychologist in the Iowa business college. I met him at the SIOP
conference in 1982. Then Duane told me about a faculty position that was open at the University of
Iowa and urged me to apply for it. I told him I was happy at OPM and wasn’t looking for another job.
So then he said, “Why don’t you just come out and visit us and give a talk. I’ll take you pheasant
hunting.”
Well, that was an offer I couldn’t refuse. I flew out there with my hunting boots and 12 gauge
Browning side-by-side shotgun. In the good old days, taking a gun with you on an airplane was not a
problem. When I got there some people I met with asked me if this were my first visit to Iowa. I said,
“No, back in 1970 I visited the Corn Palace in Mitchell, Iowa.” They looked astounded and quickly shot
back, “Frank, that’s in South Dakota, not Iowa!” It turned out I’d never been to Iowa.
I gave my talk on validity generalization and selection utility and then we went hunting the next
day, starting at 8:00 in the morning. We hunted through waist-high water grass at the Hawkeye
wildlife area until noon, and then Duane said he was too tired to continue on that afternoon. He
called Pete Schoderbek, another professor, and Pete came out and hunted with me the rest of the
day. Duane went home and took a nap.
I got my limit of three pheasants and took them back to the Holiday Inn and picked and gutted
them there in the bathtub. I put the feathers and guts in a large black plastic trash bag and put it in
the trash can in front of the hotel.
Then I got a call from Duane and his wife Sue. They invited me to their house for dinner. I took the
cleaned birds with me to get them to freeze the birds for my return trip to Washington. They were
Schmidt SIOP 2015 Living History Interview
6
expecting that we’d clean them at their house. They could not believe I had cleaned them in the
Holiday Inn bathtub! This story and the story of how I wore out two professor hunters got around the
University of Iowa business school. I like to think they concluded had physical endurance and was
resourceful.
The following year, 1983, I got another invitation to come out again and do the same thing over
again. I accepted, because by then I thought pheasant hunting might be one of my vocations in life. I
had another great pheasant hunting experience that year. I didn’t agree to accept their job offer but I
think they could see I was weakening. Then another invitation in fall 1984, another trip there, more
pheasant hunting, and I finally accepted their job offer. I started in fall 1985.
That’s how I got to the University of Iowa: recruitment through pheasant hunting. I later heard
that Duane got an award from the Human Resources Management Society for creativity in college
faculty recruitment and hiring. Duane later became the head of our department, the Management
and Industrial Relations Department. Everyone agreed he was extremely slow in answering memos,
implementing things we’d agreed on, and making decisions for the department. There was always a
lot of delay.
Duane retired in 1997, and at his retirement party I presented a tribute to his accomplishments. I
pointed out that he deserved credit for creating and being the first to use the system of recruiting
faculty via pheasant hunting, and that this method had since been introduced at many other
universities and was being widely used.
Duane was also a management theorist. Then as now, many systems of management were being
advocated—management by objectives, management by charismatic leadership, management by
employee participation, management by consideration and structure, and many others. But I pointed
out that Duane had developed a new and unique system and philosophy of management:
Management By Delay. I also revealed that he was writing a book on this new management
philosophy but that the book had been delayed. Everyone was in stitches except Duane. He knew it
was all true.
Duane still lives here in Iowa City. I visited him in early 2014. He still had not finished that book.
•
In your career you have been a professor, expert witness, author, advisor, and practitioner.
Which did you enjoy the most? Which did you enjoy the least?
My answer might surprise you. What I enjoyed the most was the challenge of trying to figure on
the epistemological riddle of how to get valid cumulative knowledge from empirical data. Jack Hunter
and I did not see meta-analysis as a mere statistical exercise. We saw it as a path to an improved
epistemology in research—a successful and superior way to attain cumulative knowledge and
establish general scientific principles in spite of the surface variability in individual study findings and
in spite of the confusion created by the nearly universal reliance on statistical significance testing in
the analysis of research data.
This was a continuation of the view that inspired our earlier pre-meta-analytic work. The basic
question was always: what do data really mean and how can we extract reliable and valid knowledge
from data? (cf., Schmidt, 1992; 2010). This included our work promoting use of confidence intervals
over significance tests (Schmidt & Hunter, 1997; Schmidt, 1996), the detection and calibration of
Schmidt SIOP 2015 Living History Interview
7
moderator variables (Schmidt & Hunter, 1978), the problem of instability of regression weights in data
analysis (Schmidt, 1971; 1972), the chance frequency of racial differences in test validity (Schmidt,
Berner, and Hunter, 1973; Schmidt, Hunter, & Hunter, 1979), and other work in this vein. So our view
of meta-analysis was that it was a continuation of this epistemological quest.
Of course, other things were enjoyable too. I had wonderful experiences teaching and mentoring
PhD students—and I always tried to include them in this epistemological quest. I think that energized
and motivated them. I was very fortunate to have so many excellent PhD students. Consulting was
also enjoyable—and broadening. My consulting work kept me focused on the idea that the work we
do should have practical, real world applications.
Schmidt SIOP 2015 Living History Interview
8
Life as a Professor
•
•
•
With over 200 publications, your career has been far more productive than most I/O psychology
professors. You also mentored some of the great names in the personnel selection field (Mike
McDaniel, Ken Pearlman, Deniz Ones, “Vish” Viswesvaran), you did consulting work, served as an
expert witness, and have a wife and three kids. How did you do it all? What was your typical
day like as a professor?
What words of wisdom do you have to new academics?
What advice do you have for I/O psychologists who aspire to achieve the success you have had?
My advice to new academics and other I/O psychologists is: Pick something to work on that you
are really passionate about. This will be highly variable between people. As I indicated, for me this
was the question of how we can obtain reliable and valid knowledge from empirical data. Because of
my methodological training, I saw that many conclusions being drawn from data in published studies
were false, and that is what stoked my interest in this question.
If you pick a problem you are really passionate about, then your work becomes you hobby—and
you are always highly motivated because it is so much fun. I did this, and looking back now it seems I
have spent my whole career working on my hobby.
What was my typical day like? There was no typical day. Every day was different. But one thing I
learned was that it was essential to spend extended periods of time without interruptions. That is
essential to both the development of ideas and the writing of research papers. An hour or two here
and there will not do it. So I would try to schedule all meetings for one day and try to leave the
following day for uninterrupted work.
Meta-Analysis
•
•
•
•
•
•
Perhaps every I/O psychologist has either conducted a meta-analysis, learned how to do one, or
read one in a journal at some point in their careers. You were one of the pioneers of this
methodology.
What inspired you to work on in this area?
One of your major contributions has been the incorporation of corrections for measurement
error and range restriction in meta-analyses. Can you tell us how you initially came up with this
idea?
To me, one of your most striking findings is the validity generalization of certain selection
devices, especially those measuring general mental ability. Were you surprised when you found
these results?
You’ve mentioned in the past that there was initially some resistance to validity generalization?
What type of resistance did you run into? What comments did reviewers make about metaanalysis and validity generalization?
Let’s play a short clip. [Play Validity Generalization Commercial]. Can you tell us about this
video?
Developing Validity Generalization and Meta-Analysis
Schmidt SIOP 2015 Living History Interview
9
It was during my time at OPM that I, along with Jack Hunter, developed our validity generalization
and meta-analysis methods—the things for which we are both best known today. This was a time of
turmoil in personnel selection because court challenges to hiring procedures under the 1964 Civil
Rights Act were at an all time high. One OPM hiring test went all the way to the Supreme Court,
where it was upheld. As a result, it was very important for OPM to have research evidence to defend
its selection tests in court. I was asked to conduct and direct such research.
Legal defense of hiring tests consisted mostly of criterion-related validity studies. One of the first
publications from our OPM research effort showed that the average statistical power of such validity
studies was only about 50% (Schmidt, Hunter, & Urry, 1976). This meant that even if a test was very
valid (a good predictor of job performance), you had only a 50-50 chance of detecting that fact in your
study.
This meant that it was very risky for an employer to conduct such a study, because if the validity
did not show up as statistically significant (that is, was not detected), the employer’s own study could
be used against that company in court. When the Ford Motor Company saw our study, they
suspended all their test validity studies—for this very reason. Much larger sample sizes were needed
to overcome this problem and increase statistical power.
Since the 1920s, industrial-organizational (I-O) psychologists had believed that test validities were
situationally specific, meaning that separate validity studies had to be conducted in each organization
and for each job. Two pieces of evidence were cited in support of this belief: The fact that statistically
significant validities showed up in only about half of all studies, and the fact that the actual magnitude
of validity estimates varied widely across studies—even when the jobs in questions were very similar
or identical.
Our statistical power article showed why the 50% statistical significance rate could not be used as
evidence supporting the situational specificity theory. It was just due to power too low to detect the
validity. But we still had to address the second piece of evidence cited in support of the situational
specificity theory. This challenge led to the development of our meta-analysis methods.
In addition to my OPM job I had an appointment at nearby George Washington University (GWU).
One day in 1975, sitting in my GWU office and pondering several highly variable distributions of
validity coefficients, I remembered an experience I had had in graduate school. As a PhD student at
Purdue, I took courses from Joseph Tiffin and Hubert Brogden. Tiffin did a lot of consulting work doing
validity studies in private industry and he reported in class that in his experience validity findings were
highly variable and seemed to depend on peculiarities of the local business situation. This was also
the position taken in textbooks and articles.
Brogden had been the Technical Director of the U.S. Army Research Institute (ARI). The validity
results he reported in class were quite stable and were very similar from study to study. So I asked
Brogden why there was this difference. He said the answer was mostly sampling error. The army had
very large samples of soldiers (often 10,000 or more in a single study) so there was little sampling
error in the results, while in industry the number of people was usually small--often 50, 70, or 100—
and there was a lot of sampling error. Sampling error causes study results to randomly jump around
all over the place. Sampling error is caused by the fact that small random samples of people are not
representative of the larger population; they do no not accurately mirror the larger population of job
applicants that you are interested in.
Schmidt SIOP 2015 Living History Interview
10
I had completely forgotten about what Brogden had said until I faced the problem of the
"bouncing validities" at OPM. Like all I-O psychologists, I had been indoctrinated with the idea of
situational specificity of validity, but when I was looking at all the variability in validities, I
remembered Brogden's comment and decided to see if I could figure out a way to measure the
amount of sampling error in a distribution of validities--that is, a way to calculate the amount of
variability expected from sampling errors and then subtract that amount from the observed variability
of the validity coefficients. I found that most of the variability—70% to 80%--was sampling error.
After subtracting out sampling error variability and applying the usual corrections for
measurement error in the job performance measures and for range restriction on ability test scores,
the typical distribution of validities had a substantial average of about .50 or so and a standard
deviation (SD) of around .10 to .15. This meant that virtually all of the actual validity values were in
the positive range and therefore it demonstrated that practically useful validities were generalizable
across situations.
When I called Jack Hunter about my results, he was positive and enthusiastic. This was reassuring
to me because these results were so surprising that I was worried that I might have made some
mistake.
It was surprising enough that GMA test validity based on the same job generalized across settings,
companies, time periods, etc. But what was even more surprising was the subsequent finding that
validity generalized across different jobs—jobs with very different task composition (Schmidt, Hunter,
& Pearlman, 1981). Ultimately, we concluded that GMA (general mental ability) tests had validity for
all jobs in the U.S. economy, with validity increasing with job complexity. Later research revealed
generalizable validity for other types of predictors too.
Jack and I refined our procedures into general meta-analysis methods to be used in personnel
selection. In late 1975 we wrote our findings up but did not immediately submit the paper for
publication, because we wanted to enter it into a research contest sponsored by the industrialorganizational psychology division of APA (Division 14). One requirement was that entries could not
already have been published.
We won that award, but as a result our first meta-analysis article was not published until 1977
(Schmidt & Hunter, 1977). Gene Glass at the University of Colorado published his first article on metaanalysis in 1976. We were not aware of Glass’ work at that time (and he was not aware of ours), but
we later realized that if had we not delayed publication so we could enter the research award contest
we could have tied Glass for the first published application of meta-analysis.
At about that time I was asked to be the OPM liaison to a National Research Council committee on
selection testing issues. One of the most famous research psychologists in the U.S., Lee J. Cronbach,
was on the committee. When I showed him our results, he told us about the work of Gene Glass. He
also stated that we should expand application of our meta-analysis methods beyond personnel
selection to psychological research literatures in general, and to do this we should write a book
describing our methods.
We published that book in 1982 (Hunter, Schmidt, & Jackson, 1982). At this time we also adopted
Glass’ term “meta-analysis” to describe these procedures; up to then we had used only the term
Schmidt SIOP 2015 Living History Interview
11
“validity generalization”. But again, Glass got there first. The meta-analysis book by Glass, McGaw,
and Smith came out in 1981.
We initially thought of our meta-analysis methods as just a solution to the problem of validity
generalization. But Lee J. Cronbach told us we should be thinking of wider applications, and we
realized that the methods had many potential applications in other areas of I-O psychology and in
many areas of general psychology. However, at that time we did not realize how widely the methods
would eventually be applied or that they would come to be used in disciplines beyond psychology. But
by the mid 1980s this was already happening. The methods were being applied widely in a variety of
different areas in I-O psychology, management research, human resources, marketing, information
sciences, finance, economics, political science, and other areas. An overview of the impact in I-O
psychology up to 2010 is presented in DeGeest and Schmidt (2011).
Confronting the Opposition to our Research
Ironically, the one area in which there was strong resistance to meta-analysis was in personnel
selection, the area for which we had originally developed the method. Our methods were not only
accepted but enthusiastically embraced in research on all relationships except the relationship
between hiring procedures and job performance.
The resistance was from both practitioners and the government agencies responsible for
administration of the 1964 Civil Rights Act. OPM accepted and embraced our findings, which put OPM
in conflict with these other government agencies. (One thing I learned during my time in Washington
is that different federal agencies are often in conflict with each other. There is no such thing as the
federal government. )
Many I-O psychology practitioners made a living conducting validity studies for employers in
business and industry. Our meta-analytic validity generalization studies showed that these expensive
situationally specific validity studies were not necessary. Our findings showed that validity generalized
across settings and jobs without any need for such studies.
The so-called “enforcement” agencies—the Equal Employment Opportunity Commission, the
Office of Federal Contract Compliance, and the Civil Rights Division of the Department of Justice—also
resisted these new findings, on grounds that these findings made it easier for employers to
demonstrate the validity of their selection methods on which some minorities tended to score less
well on average than others. They just ignored the large body of research showing that these
measures predict performance just as accurately for minorities as for others.
To this day, these agencies have refused to revise and update the government’s Uniform
Guidelines on Employee Selection Procedures (UGLs) that were issued way back in 1978, before the
newer research findings were available. This has occurred despite appeals from the Society for
Industrial and Organizational Psychology (SIOP) and other professional and scientific groups stating
that revision and updating are sorely needed.
The UGLs are not only out of date, they are inconsistent with current professional standards. This
is a much longer term example of what we have today with the global warming deniers and evolution
deniers: If you don’t like the research evidence, just pretend it doesn’t exist. Some I-O practitioners
Schmidt SIOP 2015 Living History Interview
12
have been happy with these agencies’ position, because it meant they could continue to get paid for
conducting unneeded validity studies.
Although the UGLs have not been updated, professional standards have been. The Standards for
Psychological Testing, published by the American Educational Research Association (AERA), the
American Psychological Association (APA), and the National Council on Measurement in Education
(NCME), incorporated new meta-analytic findings on validity starting in the 1990s and continuing in
the most recent edition (AERA, APA & NCME, 1985, 1999, and 2014).
The same is true for the Principles for Employee Selection, published by SIOP (Society for Industrial
and Organizational Psychology, 2003). There had initially been some resistance from professional
societies to meta-analytic findings, but this changed as the research supporting these findings piled
up.
In addition to the cumulative impact of the increasing number of meta-analytic validity studies
being published, three other factors contributed to eventual acceptance. First, the illogic of a situation
in which our meta-analysis methods were embraced and lauded for application in all areas of research
except the one area of personnel selection became apparent. People saw that this did not make sense.
The second big factor was the publication of the article “Forty Questions about Validity
Generalization and Meta-Analysis” (Schmidt, Hunter, Hirsh, and Pearlman, 1985). Jack Hunter and I,
along with Hannah Hirsh (now Rothstein), Kenneth Pearlman, Jim Caplan, Michael McDaniel, and
others at OPM who were involved in this work, had collected a long list of criticisms of meta-analysis
that we had heard expressed and we had written responses to each. [As an example, one criticism
contended that sampling error was only a hypothesis, not a fact (!).]
Milton Hakel, then the editor of Personnel Psychology, a top tier research journal, invited us to
publish these 40 questions and our answers. He also invited a group of eminent psychologists to
respond to our answers, and we to their responses. The result was the longest article ever published
by that journal—over 100 pages. This article apparently successfully addressed most or all the
reservations people had had about validity generalization and meta-analysis, because the level of
opposition dropped dramatically after that.
The last turnaround factor occurred in 1994, when the American Psychological Association
awarded Jack Hunter and me the Distinguished Scientific Contribution Award for Applications of
Psychology for our meta-analysis work. This is probably the most prestigious research award in
psychology. After that, there was really no significant opposition.
This process of acceptance took nearly 20 years. Jack and I often wondered whether we would
have done this work if we had known in advance that it would take a nearly 20 year battle to finally
get it recognized and accepted.
Jack Hunter
•
Throughout your career, you collaborated with Jack Hunter. How did you meet?
It was during my interview at MSU in 1970 that I first met Jack Hunter. He immediately impressed
me as being exceptionally smart. After I accepted the job offer and began work at Michigan State, Jack
Schmidt SIOP 2015 Living History Interview
13
and I began collaborating on research projects. We worked together and published many articles and
books together up until his death in 2002. He was truly the smartest person I ever knew, and a good
friend to boot. His intellect was such that other faculty members were intimidated by him and
avoided working with him, but I saw him as the opportunity of a lifetime.
•
•
I’m going to play a short clip of Jack Hunter talking about his childhood. [Play Clip]. For those of
us who never met him, what was he like as a person?
How did your typical collaboration go? How did you go about writing papers and doing research
together? For much of your careers, you were at Iowa and he was at Michigan. How did you
handle collaborating long-distance in the days before email and skype?
What was Jack like as a person? He had a great, laid back personality and a lot of warmth. He was
courteous and patient, yet never afraid to voice his opinion even when it was controversial. For
example, he went on television on the McNeil Lehrer Report and defended the position that the blackwhite test score difference was real and was reflected in job performance. During this show I was in
the studio in back of the set and heard some really hostile, intolerant remarks about him from the
staff there.
Also, when Jack got interested in something, he would work on it for 24 hours straight. I have
never known anyone with his powers of concentration or energy. Sometimes he almost seemed to be
in a trance when he was working.
After I left MSU and went to OPM, Jack and I collaborated via telephone and Postal mail. This was
before email was available. We often spent hours on the phone discussing research ideas and
individual papers.
Jack did not like handling the clerical details of submitting papers, corresponding with editors, etc,
so I always took care of such tasks.
Jack did come to Washington from time to time, for conferences and meetings. Occasionally I
visited him in Michigan, too. Jack’s death in 2002 was a huge loss, and I still feel it. As befits someone
with his outstanding contributions, his obituary was published in The American Psychologist. As his
closet collaborator, I wrote his obituary (Schmidt, 2003). His background and early life were highly
unusual, and I covered this and more in the obituary.
Individual Differences
•
Over 20 years ago you were one of the signers of an editorial in the Wall Street Journal that
aimed to address misconceptions about intelligence. Do you feel that these misconceptions still
exist today? What have we learned since that time?
These misconceptions are somewhat less widespread today than earlier, but they still exist and
probably always will. This is because people are motivated believe what they want to believe, and
many people want to believe that intelligence is really not very important. Accepting what we know
about intelligence from research would be too threatening.
Schmidt SIOP 2015 Living History Interview
•
14
Your meta-analytic table showing the validity of predictors of job and training performance
[show table] is one of your main contributions to the field. Do you think that there are other
predictors out there that we haven’t discovered?
Sure, there are other predictors. But the question is whether they have a useful level of
incremental validity over general intelligence. I doubt that there are very many undiscovered
predictors that do. If there are any, I suspect they won’t be in the cognitive domain but in the broad
personality or non-cognitive domain. We have known for years now that Conscientiousness is one
such example. Integrity tests are another.
•
Much of your work has focused on individual differences. You began your career at a time when
cognitive and social psychology were in their heyday and behaviorism was still strong. Did you,
and others, encounter any resistance to your work from these groups?
For the most part, experimental social psychologists and behaviorists just ignored our work. I
think both these groups became less imperialistic over the years because of what they experienced.
When I was a PhD student in the 1960s, an experimental social psychologist told me that in 20
years to field of individual differences would disappear, that individual differences would be
explained away by the environmental treatment conditions studied in their lab experiments. Well, as
you know, that did not happen. Today the field of individual differences is thriving, and experimental
social psychology is in a state of crisis, with questions being raised about their methods and the
replicatability of their findings, as I will discuss later.
As for behaviorism, it has mostly faded away, and has been replaced by cognitive psychology. At
its beginning in the 60s and 70s, cognitive psychology focused only on trying to find general laws of
cognitive processes, but in recent years the field has developed more understanding of the
importance of individual differences in cognitive processing. One example is their discovery that
working memory varies across individuals and that it is highly correlated with traditional measures of
intelligence. Two people who have contributed to this advance are Zach Hambrick (MSU) and Randy
Engle (Ga. Institute of Technology).
I think cognitive neuroscience has also contributed to this improvement. Their research has found
that fMRI scans show large individual differences and these differences are correlated with traditional
measures of IQ. The Great Courses lecture series by Richard Haier on these developments is very good
and I recommend it.
Miscellaneous
•
You have been serving as a reviewer for Personnel Psychology for 40 years (since 1975). In your
eyes, how has our field’s journals changed over that time period?
In the 1960s our journals were very atheoretical—they were dominated by blind empiricism. One
good example of this was biodata research. Along with others, I advocated a more theoretical
approach. We needed to develop theories—but theories about things with important effects in the
world of work.
Schmidt SIOP 2015 Living History Interview
15
Today I think the role of theory has been pushed too far, as pointed out a few years ago in the
now famous Donald Hambrick (2007) article. It is now virtually impossible to get an empirical article
accepted in Personnel Psychology or JAP unless it tests some currently popular theory—no matter
how good the data in the study are. A paper by my co-authors and me was rejected by JAP with the
explanation that it tested a theory other than the currently popular one.
Second, many of the theories tested are “so what?” theories, because the effect sizes they
produce are so small as to have little or no applied implications. Often this fact is not obvious because
effect sizes are not reported, only p values. Recently I had a conversation with Ben Schneider and we
found we were in agreement on this.
Another problem is that the top ranked journals strive to publish studies with novel and surprising
findings. The problem is that these are the findings least likely to replicate. Related to this finding is
the fact that, across disciplines, the higher the journal ranking, the more frequent are retractions of
articles. There is much controversy over Questionable Research Practices (QRPs). All these issues are
part and parcel of the big epistemological question I discussed earlier.
You can almost say that we have a strange situation now in which the best journals have become
the worst journals. These issues are discussed in some detail in Chapter 13 of the 2014 Schmidt and
Hunter meta-analysis book.
•
You have always called for higher standards in I/O research. What do you think are the main
areas of improvement for I/O research? What are the areas that you think we are doing well in?
Well, I won’t repeat my answer to the last question. But there are other important facts.
Today in I/O psychology, as in many other areas, the widespread use of meta-analysis has provided a
greatly improved foundation for cumulative knowledge. This is a great advance.
However, the findings of meta-analysis can be distorted by publication bias. We now have means for
detecting publication bias, and in some cases, adjusting for it.
In addition to publication bias, there are the biasing effects of questionable research practices
(QRP), as shown in recent articles by John et al. (2012) and Simmons et al. (2011) (cf. Schmidt &
Hunter, 2014, 523–530 for a detailed discussion). Examples include: (a) adding subjects one by one
until the result is significant, then stopping; (b) dropping studies or measures that are not significant;
(c) conducting multiple significance tests on a relation and reporting only those that show significance
(cherry picking); (d) deciding whether to include data after looking to see the effect on statistical
significance; (e) hypothesizing after the results are known (harking); (f) running an experiment over
until you get the “right” results. (I discovered this being done while serving on promotion and tenure
committees.)
John et al. (2012) found that large percentages of the researchers they surveyed admitted to using
these practices. Although experimental social psychology appeared to have the worst record in this
respect, it is also a problem in I/O psychology.
These problems are discussed in detail in Chapter 13 of the 2014 Schmidt and Hunter metaanalysis book. Because null hypotheses are rarely true in applied areas like I/O, the effect of QRPs
and publication is to cause overestimation of correlations and effect sizes. This is serious, because the
magnitude of effect size is very important in determining practical value and utility. The problem of
Schmidt SIOP 2015 Living History Interview
16
Type I errors is less serious, because there is much empirical evidence that the null hypothesis is very
rarely true. As of now, we do not have good methods for detecting and adjusting for the effects of
QRPs, as we do in the case of publication bias. So this is an area where further work is needed.
Retirement/Personal Life
•
•
You retired from the University of Iowa in 2012, becoming an Emeritus Professor.
What have you been doing in your retirement?
Activities since Retiring
I retired from the University of Iowa in the summer of 2012. When I retired my former PhD
students put on a big retirement party for me at the 2013 conference of the Society for IndustrialOrganizational Psychology in Houston. Over 300 people showed up for this. The food and drinks were
really great. I even enjoyed the roasting I received. I want to thank all my former PhD students for
contributing to this great event and especially Deniz Ones for being the lead organizer for this.
Since retiring, I have published the 3rd edition of my meta-analysis book, published 12 research
articles or book chapters, made 10 research presentations at conferences and universities, and
continued my consulting job with Gallup. Five other research projects are still in preparation. So far, I
have two presentations scheduled for 2015, including this interview.
Since retirement, I have received four more awards for research contributions—three national
awards and one international award. I tell people that only two things have changed with retirement:
I don’t have to teach and I don’t have to pick up my check each month! But this is the way I like it.
These things are all my hobbies, and it is a labor of love.
•
Does Frank Schmidt Bungee-Jump? (see quote below)
o http://tippie.uiowa.edu/management-organizations/schmidt.cfm: “What are your
hobbies or what do you like to do on weekends? Bungee jumping, sky diving, and hang
gliding. Also, I like to occasionally practice being a human cannon ball. (This is more fun
than you might think.)”
Frank Schmidt does not bungee jump or sky dive. But he will claim he does just to inject humor
into an otherwise overly serious discussion. Here’s the situation. A faculty member in our department
had the idea that undergraduate students should know about the private lives of their professors. She
thought this would increase their motivation. I thought this was a questionable idea. She created a
website and faculty members were required to post personal items on it. So I thought I would give the
students a story they would remember.
Schmidt SIOP 2015 Living History Interview
17
References
American Educational Research Association, American Psychological Association, and National Council
on Measurement in Education (2014). Standards for educational and psychological testing.
Washington, DC: Author. [Earlier editions were in 1985 and 1999.]
Callender, J. C., & Osburn, H. G. (1980). Development and test of a new model for validity generalization.
Journal of Applied Psychology, 65, 543–558.
Cronbach, L. J. (1975). Beyond the two disciplines of scientific psychology revisited. American
Psychologist, 30, 116–127.
DeGeest, D., & Schmidt, F. L. (2011). The impact of research synthesis methods on
Industrial/Organizational Psychology: The road from pessimism to optimism about cumulative
knowledge. Research Synthesis Methods, 1, 185–197.
Francis, G. (2012). The psychology of replication and replication in Psychology. Perspectives on
Psychological Science, 7, 585–594.
Gergen, K. J. (1982). Toward transformation in social knowledge. New York: Springer-Verlag.
Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.
Hambrick, D. C. (2007). The field of management’s devotion to theory: Too much of a good thing?
Academy of Management Journal, 50, 1346–1352.
Harter, J. K., Schmidt, F. L., & Hayes, T. L. (2002). Business unit level relationships between employee
satisfaction/engagement and business outcomes: A meta-analysis. Journal of Applied
Psychology, 87, 268–279.
Hedges, L. V. (2009). Statistical considerations. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.),
Handbook of research synthesis and meta-analysis (2nd Ed.) (pp. 37–48). New York: Russell Sage.
Hunter, J. E. (1983). Test validation for 12,000 jobs: An application of job classification and validity
generalization to the General Aptitude Test Battery (GATB). Test Research Report No. 45.
Washington, D.C.: U.S. Department of Labor, U.S. Employment Service.
Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternative predictors of job performance.
Psychological Bulletin, 96, 72–98.
Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research
findings. Thousand Oaks, CA: Sage.
Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research
findings (2nd Ed.). Thousand Oaks, CA: Sage.
Hunter, J. E., Schmidt, F. L., & Hunter, R. F. (1979). Differential validity of employment tests by race: A
comprehensive review and analysis. Psychological Bulletin, 86, 721–735.
Schmidt SIOP 2015 Living History Interview
18
Hunter, J. E., Schmidt, F. L., & Jackson, G. B. (1982). Meta-analysis: Cumulating research findings across
studies. Beverly Hills, CA: Sage.
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research
practices with incentives for truth telling. Psychological Science, 23, 524–532.
Matt, G. E., & Cook, T. D. (2009). Threats to the validity of generalized inferences. In H. Cooper, L. V.
Hedges, & J. C. Valentine (Eds.), Handbook of research synthesis and meta-analysis (2nd Ed.). (pp.
537–560). New York: Russell Sage.
McDaniel, M. A., Whetzel, D. L., Schmidt, F. L., & Mauer, S. (1994). The validity of employment
interviews: A comprehensive review and meta-analysis. Journal of Applied Psychology, 79, 599–
616.
Ones, D. S., Viswesvaran, C., & Schmidt, F. L. (1993). Comprehensive meta-analysis of integrity test
validities: Findings and implications for personnel selection and theories of job performance.
Journal of Applied Psychology Monograph, 78, 679–703.
Orlitzky, M., Schmidt, F. L., & Rynes, S. L. (2003). Corporate social and financial performance: A metaanalysis. Organizational Studies, 24, 403–441.
Pearlman, K., Schmidt, F. L., & Hunter, J. E. (1980). Validity generalization results for tests used to predict
job proficiency and training criteria in clerical occupations. Journal of Applied Psychology, 65,
373–407.
Raju, N. S., & Burke, M. J. (1983). Two new procedures for studying validity generalization. Journal of
Applied Psychology, 68, 382–395.
Raju, N. S., Burke, M. J., Normand, J., & Langlois, G. M. (1991). A new meta-analysis approach. Journal of
Applied Psychology, 76, 432–446.
Rubin, D. (1990). A new perspective on meta-analysis. In K. W. Wachter & M. L. Straf (Eds.), The future of
meta-analysis. New York: Russell Sage (pp. 155–166).
Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple study articles.
Psychological Methods, 17, 551–566.
Schmidt, F. L. (1969). Distortion of drivers’ estimates of automobile speed as a function of speed
adaptation. Journal of Applied Psychology, 53, 536–539.
Schmidt, F. L. (1971). The relative efficiency of regression and simple unit predictor weights in applied
differential psychology. Educational and Psychological Measurement, 31, 699–714.
Schmidt, F. L. (1972). The reliability of differences between linear regression weights in applied
differential psychology. Educational and Psychological Measurement, 32, 879–886.
Schmidt SIOP 2015 Living History Interview
19
Schmidt, F. L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative
knowledge in psychology. American Psychologist, 47, 1173–1181.
Schmidt, F. L. (1996). Statistical significance testing and cumulative knowledge in psychology:
Implications for the training of researchers. Psychological Methods, 1, 115–129.
Schmidt, F. L. (2003). John E. Hunter, 1939–2002. Obituary. American Psychologist, 58, 238.
Schmidt, F. L. (2010). Detecting and correcting the lies that data tell. Perspectives on Psychological
Science, 5, 233–242.
Schmidt, F. L. (in press). History and development of the Schmidt-Hunter mea-analysis methods.
Research Synthesis Methods. (to be published in 2015)
Schmidt, F. L., Berner, J. G., & Hunter, J. E. (1973). Racial differences in validity of employment tests:
Reality or illusion? Journal of Applied Psychology, 58, 5–9.
Schmidt, F. L., & Hunter, J. E. (1977). Development of a general solution to the problem of validity
generalization. Journal of Applied Psychology, 62, 529–540.
Schmidt, F. L., & Hunter, J. E. (1978). Moderator research and the law of small numbers. Personnel
Psychology, 31, 215–232.
Schmidt, F. L., & Hunter, J. E. (1997). Eight common but false objections to the discontinuation of
significance testing in the analysis of research data. In L. Harlow, S. Muliak, & J. Steiger (Eds.),
What if there were no significance tests? (pp. 37–64). Mahwah, NJ: Lawrence Erlbaum. Schmidt,
F. L., & Hunter, J. E. (2014). Methods of meta-analysis: Correcting error and bias in research
findings (3rd Ed.). Thousand Oaks, CA: Sage.
Schmidt, F. L., Hunter, J. E., Outerbridge, A. M., & Trattner, M. H. (1986). The economic impact of job
selection methods on the size, productivity, and payroll costs of the federal work-force.
Personnel Psychology, 39, 1–29.
Schmidt, F.L., Hunter, J. E., & Pearlman, K. (1981). Task differences and the validity of aptitude tests in
selection: A red herring. Journal of Applied Psychology, 66, 166–185.
Schmidt, F. L., Hunter, J. E., Pearlman, K., & Hirsh, H. R. (1985). Forty questions about validity
generalization and meta-analysis. Personnel Psychology, 38, 697–798.
Schmidt, F. L., Hunter, J. E., Pearlman, K., & Shane, G. S. (1979). Further tests of the Schmidt-Hunter
Bayesian validity generalization model. Personnel Psychology, 32, 257–281.
Schmidt, F. L., Hunter, J. E., & Urry, V. E. (1976). Statistical power in criterion-related validation studies.
Journal of Applied Psychology, 61, 473–485.
Schmidt, F. L., & Oh, I-S. (2013). Methods for second order meta-analysis and illustrative applications.
Organizational Behavior and Human Decision Making, 121, 204–218.
Schmidt SIOP 2015 Living History Interview
20
Schmidt, F. L., Oh, I-S., & Hayes, T. L. (2009). Fixed versus random models in meta-analysis: Model
properties and comparison of differences in results. British Journal of Mathematical and
Statistical Psychology, 62, 97–128.
Shadish, W. R. (in press). The meta-analysis big bang. Research Synthesis Methods. (To be published in
2015)
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in
data collection and analysis allows presenting anything as significant. Psychological Science, 22,
1359–1366.
Society for Industrial and Organizational Psychology (2003). Principles for the validation and use of
personnel selection procedures (4th Ed.). Bowling Green, OH: Author.
Whetzel, D. L., McCloy, R. A., Hooper, A., Russell, T. L., Waters, S. D., Campbell, W. J., & Ramos, R. A.
(2011). Meta-analysis of clerical performance predictors: Still stable after all these years. Journal
of Selection and Assessment, 19, 41–50.