Good Science–Good Practice
Jamie Madigan
Ameren
Marcus W. Dickson
Wayne State University
In the inaugural issue of the Good Science–Good Practice column (TIP,
April 2006), we discussed how we would be on the lookout for noteworthy
research and publications that effectively bridge the scientist/practitioner
divide. It just so happens that that the 2006 SIOP conference meant that we
didn’t have to look much further than Dallas, Texas for a remarkable batch of
papers that match that criteria pretty well. These aren’t the only presentations
and papers that do justice to the good science–good practice framework
(there was too much great stuff to see, let alone write about), but they are
some of the ones that grabbed our attention more tightly.
Unproctored Internet Testing: What Do the Data Say?
(conference session 25)
One noteworthy symposium dealt with unproctored, Internet-based
employment testing. Although this has been a hotly debated topic at SIOP and
elsewhere in the last few years, this symposium, entitled “Unproctored Internet Testing: What Do the Data Say,” caught our attention. The main reason is
that it included the results of some impressive research to back up speculations about what happens when companies move to a testing program that lets
applicants take the test “at two-thirty in the morning in their bathrobe.”
Jana Fallon, Jay Janovics, Jason Read, James Beaty, Eyal Grauer,
and Josh Davis presented empirical research looking, among other things, at
the effects of moving testing up to the beginning of the selection process and
making the tests unproctored. Generally, test score means were stable and did
not change over time.
However, in perhaps what was the most eye-opening presentation, Ken
Lahti and Paul DeKoekkoek discussed their utility analysis and ROI estimates for unproctored testing. What made this discussion interesting was that
it pointed directly at the elephant in the room: Most unproctored testing programs omit a cognitively loaded test, and when you do that, the validity of
your overall process diminishes. Even with conservative estimates of the dip
in validity, the researchers found that if the move to unproctored testing
means dropping cognitive tests, the utility of the whole system does not compare favorably to the utility of proctored tests that do contain the cognitive
The Industrial-Organizational Psychologist
73
component. It’s a great example of how rigorous research can reveal problems when real-life changes are made to a selection system, as well as suggest how they can be circumvented (e.g., doing part of the testing unproctored and finishing it off with a proctored session with fewer test takers).
Learn ’N’ Play: Effectiveness of Videogame-Based Simulations for
Training and Development (conference session 153)
Keeping with the digital theme, one of the more inherently interesting symposia dealt with the application of video games to enhancing training effectiveness. The session contained two theoretical pieces and two empirical studies,
which fits the needs of this column almost by definition. Karin Orvis, Michael
Garrity, Diane Miller, Janis Cannon-Bowers, Ben Sawyer, and Adams Greenwood-Ericksen made presentations that looked at where the nature of video
games might provide benefit to (or detract from) training effectiveness. By
sheer virtue of their interactivity, engagement, and complexity, videogames represent a chance for trainers to draw trainees in and get them to meet learning
objectives—if done correctly. The presentation by Cannon-Bowers in particular was an interesting summary of an entire research line that at the moment is
practically untapped by practitioners. It’s a blank slate ready to be filled in.
James Belanich, Laura Mullin, Karin Ovis and Daniel Horn supplemented
the theoretical pieces with research looking at how the effectiveness of
videogame-based training is influenced by both characteristics of the game
and characteristics of the trainee. Belanich and Mullin, for example, provided
useful data for researchers seeking to maximize the use of videogame-based
training by finding that training-related information tended to be recalled more
successfully when the videogame presented it with spoken text or graphical
images. Likewise, information was recalled more frequently if it was related
to the context of the videogame, such as in mission details or game objectives.
Orvis, Belanich, and Horn also presented a study whose findings suggest that
trainee characteristics such as familiarity with videogames and goal orientation could influence the effectiveness of such training.
Again, these represent tantalizing starting points for fascinating research
at the intersection of science, HR systems, and entertainment. Audience discussion inevitably turned to the use of videogames for employment testing
and even exercises to measure emergent leadership. We look forward to more
research in this vein.
Testing Strategies for Reducing Adverse Impact
(conference session 54)
As a field, we’ve wrestled with issues of bias and differential selection rates
for years. This symposium presented several new approaches to these problems, and though much of what was presented is still in developmental phases,
74
July 2006
Volume 44 Number 1
practitioners in selection and assessment should be aware of these developments that are coming down the pike. Anu Ramesh, Paul Hanges and Michael
Dougherty’s paper on “Measuring Working Memory in Firefighter Applicants:
Validity and Adverse Impact” focused on “working memory,” which includes
things like allocating resources, actively holding information for access, and
processing information, and which differs from short-term memory in that
working memory is active memory rather than simply storage. They found that
a test of working memory was as reliable as more traditional tests of reading
and math abilities, that working memory was significantly correlated with
those traditional tests (r = .39), and that working memory had lower adverse
impact against Blacks than did the traditional tests (48 s vs. .78 s), when
applied to a sample of applicants for entry-level firefighting positions.
In “Black–White Differences in Reading Comprehension: The Measure
Matters,” Mina Sipe, Paul Hanges, and Harold Goldstein presented evidence that a new reading test—the Reading Components Processing Test
(RCPT), developed by Hannon and Daneman (2001)—correlates significantly with standard reading comprehension tests but with reduced adverse
impact. The RCPT includes four constructs: text memory, text inferencing,
knowledge access, and knowledge integration. The test did correlate with traditional reading comprehension tests and in some cases showed significantly
reduced adverse impact against Blacks, but unexpectedly, it also showed (for
one construct) significant adverse impact against Whites.
Finally, in “Does Relaxing Time Limits on Cognitive Ability Tests
Reduce Adverse Impact? An Examination of Self-Regulation as a Moderator,” David Mayer, Lili Duan, and Paul Hanges presented evidence that the
elimination of time limits on cognitive ability tests can reduce adverse
impact, but consistent with self-regulation theory, the effects were primarily
driven by improved performance by those Black test takers who were low on
time management and self-regulation skills.
All of these presentations demonstrate that I-O psychologists are continuing to wrestle with test bias and adverse impact and that creative new
approaches to research will continue to provide practitioners with better tools
to select talented people while reducing the influence of nonrelevant variance.
Obviously, there were several excellent symposia at the conference. But
we also hit the poster sessions and wanted to mention a couple of the posters
that caught our attention, as well.
Social Responsibility, Ethical Leadership and Effectiveness
(conference session 211-9)
Annebel De Hoogh and Deanne Den Hartog used a sample of over 70
CEOs and two separate samples of direct reports of those CEOs. They found
that leaders who were perceived by direct subordinates to be high on personal commitment to social responsibility (measured unobtrusively, and incorThe Industrial-Organizational Psychologist
75
porating moral–legal standards of conduct, inner obligation, concern for others, concern for negative consequences and self-judgment) were seen as more
ethical leaders. More importantly, those leaders rated by one group of direct
reports as being ethical tended to have top management teams that were rated
(by a second group of direct reports) as being more highly effective. The top
management team members working for those leaders were also more likely
to say that they saw a place for themselves in the organization and to endorse
other statements that reflected a low intention to turnover. All of these findings together seem to tie in with the growing literature on ethical climate and
the leader’s role in the creation of ethical climate—but the use of unobtrusive
measures to assess the leaders’ patterns of values, the use of two separate
groups of direct reports (so as to avoid same-source bias for any of the ratings), and the use of a relatively large and diverse sample of CEOs all contribute to this paper’s advancement of our understanding of the psychological phenomenon involved in ethical leadership and ethical climate. They also
point to specific and tangible outcomes of ethical leadership and commitment
to social responsibility at the top of the organization—something that organizational practitioners should be most interested in.
National Culture Compatibility and Merger and Acquisition Performance
(conference session 52-12)
Jumping several levels of analysis, we found the research by Vasiliki
(Kiki) Nicolopoulos and Harold Goldstein to be a strong example of combining existing data sets to make something new and very practically useful.
Nicolopoulos and Goldstein used the cultural values scores from the GLOBE
Project data set, along with data from the stock market and other financial
indicators of merger and acquisition performance. Their hypotheses were
essentially that organizations from cultures with relatively similar dominant
work values would be more easily integrated in either a merger or an acquisition than would organizations from cultures with radically dissimilar patterns
of work values. This seemed likely, based on the GLOBE findings that in general, the values shared within organizations tend to reflect the values shared
within the societies in which those organizations originate. Of course, there are
many different ways of assessing the success of international M&A, with
some reflecting short-term gains and others having a more long-term focus,
and rarely do these various measures all tell the same story. In this case, the
study’s results showed that the effects of “culture clash” varied depending on
which dimensions of culture were assessed, with both power distance practices and institutional collectivism values differences between merging firms
unexpectedly correlating negatively with market adjusted returns 1 year later.
In addition, there were main effects of industry (the success of manufacturing
M&As was 19 times lower than for service M&As, based on a measure of
short-term raw returns), as well as interactions between culture clash and
76
July 2006
Volume 44 Number 1
industry. All told, the study shows that it is worth paying attention to culture
differences between firms engaging in mergers and acquisitions across national borders but that the effect of culture clash is quite complex.
As you can see, SIOP was full of presentations that push forward science
and provide important and useful information to practitioners. Next issue,
we’ll get to our more usual format of highlighting recently published articles
that fulfill both of those goals. If you want more information on any of the
research we discussed in this issue, you can search the SIOP 2006 Conference
Program online at http://www.siop.org/ProgramOnWeb/?year=2006 and contact the submitter for the relevant symposium or poster.
Keynote Speakers:
Robert Eichinger, CEO of the Lominger Limited,
Inc, a Minneapolis-based consulting firm
Leslie Joyce, vice president and chief learning
officer at Home Depot in Atlanta
William H. Macey, CEO of Valtera Corporation,
a consulting firm in Rolling Meadows, Illinois
Check www.siop.org/lec/ for the latest information
The Industrial-Organizational Psychologist
77