The Leadership Quarterly 23 (2012) 1104–1133
Contents lists available at SciVerse ScienceDirect
The Leadership Quarterly
journal homepage: www.elsevier.com/locate/leaqua
Quantifying leader lives: What historiometric approaches can tell us☆
Gina Scott Ligon a,⁎, Daniel J. Harris a, Samuel T. Hunter b
a
b
University of Nebraska at Omaha, USA
The Pennsylvania State College, USA
a r t i c l e
i n f o
Article history:
Received 14 May 2012
Received in revised form 15 September 2012
Accepted 3 October 2012
Available online 14 November 2012
Keywords:
Outstanding leadership
Life history
Historiometric
Q-Sort
Multivariate analysis
a b s t r a c t
Historically notable leaders, such as John F. Kennedy, Mohandas Ghandi, and Rupert Murdoch,
serve as exemplars in our field of study. Across the domains of military, business, government,
ideology, and the arts, these outstanding leaders have markedly impacted the institutions, fields,
and broader social structures in which they worked and lived. To understand the unique styles,
developmental experiences, and performance contributions of such leaders, a historiometric
approach is encouraged. We define this methodological strategy as the study of multiple eminent
leaders that requires the translation of historical, qualitative information into quantitative indices
of individual differences in leaders. Best practices, limitations, and implications for this technique
as an advancement of leadership theory are reviewed.
© 2012 Elsevier Inc. All rights reserved.
1. Introduction
The biographic accounts of John F. Kennedy, Mohandas Ghandi, and Rupert Murdoch portray leaders, albeit markedly different
types of leaders (Chadha, 1999; Perret, 2002; Shawcross, 1997, respectively). Accounts of each of these and other historically
notable leaders show that they had disproportionate impact on the institutions and fields in which they worked and the social
structures in which they lived (Bass, 1985), and upon close examination it becomes apparent that they did so differentially
(Mumford, 2006). Reviewing historical accounts of the early lives and careers of influential leaders also illustrates that each rose
to his respective height of power through unique patterns of life events (Ligon, Hunter, & Mumford, 2008). Post hoc examination
of their contributions to society gives some indication of who created more or less sustainable institutions, movements, and/or
policies in his or her tenure. In short, biographical and historical accounts of these leaders contain numerous objective indicators
of psychological, contextual, and performance variables and patterns.
Moreover, qualitative methods such as the examination of historical records allow us to identify leader behaviors that might
be considered less salient (e.g., problem solving) or difficult to observe via “typical leadership study” methods (Hunter,
Bedell-Avers, & Mumford, 2007). For example, a relatively new theory of leadership, the Charismatic, Ideological, and Pragmatic
(CIP) model, based on observable differences in leader cognition and sensemaking across three markedly different styles of
outstanding leadership, would not have emerged if not for historiometric methodology (Mumford, 2006). In short, transforming
qualitative information into quantitative metrics to build and test theoretical frameworks is an exciting methodology for our field
as it provides access to leader characteristics otherwise difficult to tap.
So why is it that we do not incorporate the study of these objectively-verifiable accounts of such leaders into our empirical
paradigms more systematically? The study of multiple outstanding leaders that requires the translation of historical, qualitative
☆ This material is based upon work supported by the U.S. Department of Homeland Security under Grant Award Number 2010-ST-061-RE0001. The views and
conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or
implied, of the U.S. Department of Homeland Security.
⁎ Corresponding author. Tel.: +1 402 554 2972.
E-mail address: [email protected] (G.S. Ligon).
1048-9843/$ – see front matter © 2012 Elsevier Inc. All rights reserved.
http://dx.doi.org/10.1016/j.leaqua.2012.10.004
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
1105
information into quantitative indices of individual differences in leaders, or leader historiometric analysis, is an underutilized
approach to the empirical study of leadership for at least three reasons. First, early uses of historiometric techniques drew
criticism due to the inference of psychological variables from hard, objective data (Cox, 1926). However, because more recent
operationalizations of psychological variables tend to rely upon concrete, objectively-measured behaviors (as opposed to general
traits), contemporary historiometric investigations of such psychological variables may be on the upswing (e.g., Deluga, 2001;
Winter, 1987). Second, biographic data, often gathered by historians untrained in leadership theory, are typically incomplete.
Moreover, although academic biographies tend to focus on accounts of events that can be verified by multiple parties, such as life
events (e.g., schools attended) and public career decisions (e.g., speeches given), rivate interactions (e.g., those between a leader
and her spouse) are less often reported as they cannot be verified as readily.
While who gathers the record has implications for the type of data that can be garnered from a historical biography, it does not
negate the utility of the information that can be obtained, especially when a careful content coding scheme is applied by trained
raters. Related to this is the third reason historiometric techniques have been underutilized in the field of leadership research:
they are labor intensive and complex to execute. To ensure reliable and valid inferences are drawn from the often heterogeneous
information found in historical records, a multi-step data analytic strategy is required to exam multiple cases. These steps, while
described in pieces across various domains (McKeown & Thomas, 1988; Mumford & Threlfall, 1992; Simonton, 1988a, 1990), have
not been conceptually described together in one place with regard to the study of leadership.
Despite reasons for the relative lack of historiometric studies conducted, when the research question depends on defining the
patterns of individual differences in the behaviors, developmental experiences, and performance of leaders — and particularly
outstanding leaders who serve as exemplars for marked influence and performance — we argue that a historiometric approach is
the best available methodological framework to address the issues at hand. In short, historiometric approaches offer an empirical
method to test leadership theory in a way that other approaches do not. Thus, the primary purpose of this submission is to provide a
detailed account of the use of historiometric approaches to empirically test individual differences in patterns of attributes, style, problem
solving, and performance among leaders. Prior to this, we first review a commonly employed sample — outstanding leaders —
where application of historiometry is particularly well-suited, and we describe typical frameworks used to study it.
2. Outstanding leaders
Outstanding leaders are those individuals who have had a marked influence on their organizations and on the broader social system
(Bass, 1985; Gardner, 1993). For example, the laws that govern how we work today can be credited to the Roosevelts, both Teddy and
Franklin, through their influence over the passage of fair labor laws. Two important characteristics differentiate outstanding leaders from
more normative, day-to-day leaders. First, outstanding leadership is a rare event. Because a central tenet of how we define an
outstanding leader lies within having disproportionate impact on the way we live, only few individuals can rise to positions where the
expression of such a vision for leadership influence can be realized. Second, these leaders, and the contextual influences around them
(e.g., close followers, performance outcomes), are extensively chronicled — by multiple types of sources. What is important to note is
that while seemingly positive in connotation, outstanding leaders can exert their disproportionate influence through ethical or unethical
means, a distinction that has been described as personalized versus socialized leadership (House & Howell, 1992). Multiple sources exist
detailing conditions and characteristics that give rise to both of these types of leadership styles. Implications of these points are clear:
outstanding leadership, both in positive and destructive forms, is a low base-rate phenomenon that yields insight into how to achieve
large scale outcomes (good and bad). This does not negate the applicability of such lessons to the “rest of us,” it just means that they
stand as exemplars for what leadership can be — which is highly influential and impactful.
2.1. Typical approaches used in studying outstanding leadership
Most research on outstanding leadership has been conducted through four traditional approaches: 1) dimension validation via
normalcy, 2) simulation, 3) case study, and 4) quantitative collective case analysis. The first approach, dimension validation via normalcy,
is essentially the validation of outstanding leadership qualities through generalization to normal leader samples. Put differently, the
qualities of outstanding leaders are compared against those of normal leaders, and conclusions are drawn regarding the collective
difference between outstanding leaders and normal, non-outstanding leaders (for example, see Avolio & Bass, 1999). The benefit of using
this approach to study outstanding leadership is that it helps differentiate outstanding leaders from more normative leaders. However,
because the foundation of this approach lies in comparisons between groups (i.e., normal versus outstanding leaders), a drawback to this
approach is that it does not allow within group comparisons; researchers cannot differentiate between individual outstanding leaders.
Another drawback to this approach is that the contexts in which normal leaders typically conduct their affairs may not generalize to
those contexts in which outstanding leaders conduct theirs, such as leading a small office division versus leading a country. As leadership,
like most psychological variables, operates on a continuum, it makes sense that we study extreme cases who stand outside most
leadership analyses in addition to those in more typical methodological approaches (McKelvey et al., 1999).
The second typical approach of studying outstanding leadership is to conduct simulations. Those simulations are often
achieved by giving participants various scripts based on leader behavior, the purpose of which is generally to determine how
followers respond to scripts that exemplify some outstanding leadership style. Likewise, another type of simulation is to have
participants take the roles of outstanding leaders. For an example of the latter type, see Hunter, Bedell-Avers, and Mumford
(2009). A benefit to using simulations is that, because of their highly-structured nature, they are appropriate for understanding
discrete cognitive processes. However, because of that high amount of structure and the common use of student samples, the
1106
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
results of such studies are often difficult to generalize to workforce samples and the contexts of a normal organization, which are
generally not highly structured and are more ambiguous than situations presented in simulations.
The third approach, the case study, requires analysis of a single case in significant detail, often held against a particular leadership
theory. A case study by Neal and Tansey (2010), for example, thoroughly explored the leadership of Rafik Hariri, the former Lebanese
Prime Minister. A benefit to using the case study approach is the level of description that can be gathered about one case, often from
multiple sources. Such description is important to better understand those cases, and how they offer evidence for a particular
leadership theory, at great depth. In addition, there is much research in organizational behavior (Gibbert, Ruigrok, & Wicki, 2008;
Kempster, 2006, 2009) and political psychology literature (Kaarbo & Beasley, 1999) on how to conduct such case analyses
systematically. However, in these fields such case research is typically qualitative, making it difficult to statistically compare case
studies to one another and within a larger theoretical framework. Also, because only one case is generally examined, the conclusions
drawn from the analyses are often difficult to generalize to other cases or be applied to sets of cases.
The fourth and final typical approach to studying outstanding leadership is the quantitative collective case study. This
approach builds on the case study approach in two important ways. First, quantitative case analysis typically utilizes multiple
cases instead of just one (Jauch, Osborn, & Martin, 1980). Second, and more importantly, a coding scheme is applied to the
qualitative data to obtain quantitative indices of leader behavior. A benefit of this approach is that it takes into account the context
in which each individual leader conducted his or her affairs and, because of the depth of the investigations, the data are often
quite rich. However, the conclusions and analyses can only be as good as the data afford, and rigorous controls need to be applied
to the quantitative analyses to minimize the possibility of certain biases.
The historiometric approach represents a special case of this final approach. Specifically, historiometric research is the
application of a content coding scheme to qualitative data across multiple cases, with the further implication that such qualitative
data are historically notable in nature.
3. Historiometric approach
Historiometric methods date back to some of the oldest analytic techniques in social science research (Woods, 1909, 1911),
but it has been refined over the years by Dean Simonton's social psychology research program and Mike Mumford's outstanding
leadership research program. This method relies upon available and verifiable historic records (Mumford, 2006; Simonton, 1990,
1998a, 2003). His and others' (Tetlock, 1979; Winter, 1987) work have at least three implications for leadership research. First,
this type of data comes in many forms, such as financial documents, school transcripts, and employment history, lending
credibility to the reliability and validity of the information it yields. Raw data used in these studies does not come from
laboratories or other “live human participant protocols;” instead, data used in these investigations come from historical records of
people and situations of notable importance. This last piece is important for studies of outstanding leadership: in order to obtain
multiple, historically verifiable markers of behavior and performance, the cases under examination need to have left enough of a
“mark” on history that trained historians — and multiple historians — took time to investigate and write about them. The people
publishing these works are typically not trained leadership or organizational behavior researchers. Instead, most of the sources
used in historiometric research come from academic historians and political scientists.
This is related to the second characteristic of historiometric research — it requires quantitative analysis of the qualitative
information provided in such historical documents. A multi-step data analytic technique is required for historiometry. First,
ratings must be obtained from the qualitative data at hand. Because historical records are not prepared by leadership scholars in
most cases, a clear set of variables, time frame, and type of information must be defined a priori (Mumford, 2006; Tetlock, 1979).
Second, the scores yielded from these ratings are subjected to rigorous quasi-experimental and advanced multivariate analysis.
While the data are correlational in nature, careful selection of sophisticated analytic techniques can afford more certainty around
obtained inferences about relationships (Simonton, 1990).
Third, and perhaps most important, the goal of historiometric research is to test theory (Mumford, 2006; Simonton, 2003).
Whereas quantitative single-case analysis is intended to develop theory based on detailed examination of a single case, the
multiple observations afforded in a well-designed historiometric study allow us to investigate highly complex theories of
individual differences among outstanding leaders. The aim of the next section is to illustrate the type of information relevant to
outstanding leadership theories that can be derived from a historiometric approach.
3.1. Leadership data available from historiometric approaches
There are at least four types of leadership data available through historiometry: 1) situational influences/conditions,
2) behaviors, 3) developmental events, and 4) sustainable performance. First, rich contextual data can be used to examine
boundary conditions for a given leadership style's effectiveness (Eisenhardt, 1989; Weber, 1926). For example, Tetlock (1979,
1981, 1983). Coding critical events of hundreds of leaders solving international crises, he found low cognitive complexity is
associated with taking extremist positions and high cognitive complexity is associated with more moderate approaches to conflict
resolution. In a recent study of terrorist organizations, Asal and Rethmeyer (2011) found that only leaders who sought to make
alliances across ideological and organizational lines proved to have most threat for terrorism to the United States. By assessing
individual differences embedded in the culture and organization at large, leadership scholars are better able to draw inferences
about generality and utility of our theories.
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
1107
Second, and as previously discussed, historiometric research yields data on behavioral markers of individual differences. While Cox
(1926) and Watson's (1927) original work on intelligence was the pioneering data in this area, recent approaches by Simonton (1983,
1991, 2006) have demonstrated that biographic summaries of eminent leaders can differentiate leaders on rated intelligence and
problem solving style. In addition to intelligence, a comprehensive approach in differentiating 120 notable charismatic, ideological, and
pragmatic (CIP model) leaders on aspects of the CIP model, Mumford, Bedell, Hunter, Espejo, and Boatman (2006); Mumford, Gaddis,
Licuanan, Ersland, and Siekel (2006); Mumford, Gaddis, Strange and Scott (2006); Mumford, Scott, and Hunter (2006); Mumford,
Strange, Gaddis, Licuanan, and Scott (2006) extracted behaviors indicative of vision formation, political skill, propensity for violence,
creative problem solving, and leader member exchange relationships using historiometric techniques. Other researchers have applied
this technique to measure behavioral markers of narcissism (Deluga, 1997, 1998, 2001), Machiavellianism (Bedell, Hunter, Angie, &
Vert, 2006), and networking (Bedell-Avers, Hunter, Angie, Eubanks, & Mumford, 2009) in outstanding leaders.
Third, historiometric approaches provide ample data about developmental experiences and seminal events as related to leadership
style. As the importance of experiential development has taken a front seat in contemporary leadership research (Day, 2000; Ligon &
Hunter, 2010; McCall, 2010), examining how leaders develop over time seems like a logical phenomenon to study. In other domains, this
adult development literature is abundant. For example, in examining the life histories of eminent creative scientists and artists, Simonton
(1975, 1988b). In an examination of the obituaries of leading scientists, Mumford et al. (2005) identified important career events such as
early exposure to the field and supervised practice as critical indicators of late-life innovation. Finally, in a review of early career events,
clear differences arose between the patterns of development between charismatic, ideological, and pragmatic leaders (Ligon et al., 2008).
Table 1 provides an example of the type of developmental event information yielded from historiometric approaches.
Finally, and perhaps most important, historiometric approaches allow us to examine the sustainability of performance
(Bedell-Avers et al., 2009; Mumford, 2006). Because the approach itself is rooted in the historical significance of these individuals,
examining the lasting (and sometimes non-lasting) effects of their decisions provides us with a time-tested gauge of performance. Two
examples are relevant here. First, upon examination of the most popular selling bands in 1992, the group Boys II Men clearly looks to be
the front-runner, breaking Elvis' record for number of weeks at #1 on the Billboard Top 100 Chart for their song, “End of the Road.”
Pearl Jam, another band also at their arguable “height of power” in 1992, however, continues to produce solid albums and perform to
sell-out crowds some twenty years later. Archival analysis allows us to determine whose performance was indeed more sustainable in
the long-term, albeit slightly less “popular” if performance were measured at only one juncture (Pearl Jam was on the Billboard Top
100 Chart in 1992 for two weeks, but never at #1 spot). Relevant to leadership, temporal stability of criteria also come into play. In a
series of studies examining Presidential influence, Simonton (1986a, 1986b, 1987) and Winter (1987) found that the performance
criteria of a) contemporary popularity versus b) long-term success was differentially predicted by leadership characteristics.
Thus, it is unlikely that studying more normative or run-of-the-mill leaders via contemporary empirical methods (e.g., lab
studies, survey research) will afford generalization to outstanding leader samples. However, we can learn much about leadership
as a phenomenon from studying cases of extreme leadership, or the leadership that emerges in complex, ill-defined situations and
domains (Bass, 1985). Moreover, there are processes — such as communication strategies, sensemaking, and problem solving —
that may generalize to all types of leader situations if adapted to the context at hand. For example, a high school football coach can
readily use the same techniques deployed by NCAA legends to motivate his team and coaching staff through ideological
sensegiving strategies (Hunter, Cushenbery, Thoroughgood, & Ligon, 2011). In terms of understanding formative influences, as
these leaders are deemed “outstanding” given their disproportionate influence, seeing what objective career events (e.g.,
mentoring) are related to subsequent performance may yield insight for leader development in a host of domains. What is
Table 1
Example information found in historiometric approaches.
Early life events
Idi Amin
Bill Belichick
Osama bin Laden
“In 1940 Amin went to live with a maternal uncle in Bombo and tried to register for formal schooling, but he was rejected because
Nubians could not then get admission to schools. He is said to have taken part and been injured in the Nubian anti-discrimination
riots in the same year. In 1941 he was accepted by the Garaya Islamic School in Bombo and attended until the end of 1944. At
Gayara he showed that his was a mind of potential, excelling in and winning honours for his skills in Qur'anic recitation in 1943.
After leaving he got a job as door attendant and hat and coat checker at the Imperial Hotel in Kampala. It was here that he
impressed a British officer who recruited him into the army” (Moghal, 2010, p. 9).
“The man behind the dynasty is himself the son of a football coach. Bill's father was head football coach at the Naval Academy, and
young Bill grew up idolizing Navy players such as Roger Staubach, the 1964 Heisman trophy winner, who, after his naval service,
won the Super Bowls for the Dallas Cowboys. Bill's mother said the coaches did not mind having young Bill around. ‘He wasn't a
best. He was very good at listening and learning and remembering.’ Bill spent a year in prep school and then attended Wesleyan in
Middletown, Connecticut. As reserve defensive end, he had questions: ‘What if the offense did that?’ Or ‘Whose responsibility is
that?’ He served as team co-captain of the lacrosse team. What he really gained in college was an approach to learning that was
both strategic and tactical. ‘You learn how to think and solve problems,’ he says. ‘Here's the problem; you go figure it out.’ He was
an economics major.” (Baldoni, 2005, p. 112)
“In his youth, Osama regularly met prominent Islamists in social and religious settings arranged by his father and — after
Muhammad's death — his older brothers. These events, one observer noted, ‘started forming an Islamic responsibility in him at an
early age.’” (Scheuer, 2011, p. 24)
“He also accompanied Allia and his step siblings on summer holidays in Syria for his first seventeen years, experiencing a more
cosmopolitan atmosphere than that in the Kingdom. Thus in watching his mother and traveling with her during the summers,
Osama was further acquainted — as he had been at his father's halaqat — with the diversity of the Muslim world, learning how to
get along with devout Muslims who did not embrace the Kingdom's puritanism.” (Scheuer, 2011, p. 27)
1108
Table 2
Historiometric studies involving leadership.
Leader(s)/type of leader(s)
Research question
Conclusion
Data source(s)
Main constructs of
assessment
Construct codings
Statistical analyses
Bass and Farrow (1977)
Political figures
How are leaders'
actions influenced by
situational factors?
Different leadership
styles are used
depending on situational
factors.
– Biographies
– Various documents
– Ratings
– Judgments
- Scale reliability
– Correlation
– Factor analysis
Bass, Avolio, and Goodheim
(1987)
World-class leaders
Can biographies be
used to differentiate
leadership styles of
well-known leaders?
The use of biographies
can be a valid and
reliable way to measure
leadership qualities.
– Biographies
– System inputs of
the organization,
work group, task,
attitudes, leader
styles, and within
system relations
– System outputs
– Transformational
leadership style
– Transactional
leadership style
– Ratings
Bedell et al. (2006)
Outstanding leaders in
the 20th century
Different leadership
styles and orientations
differ in levels of
Machiavellianism, and
that Machiavellianism
is negatively related to
leadership
performance.
– Highly structured
academic biographies
– Pathways to
Outstanding Leadership
– Machiavellianism
– Leadership
performance
– Leadership style
– Leadership orientation
– Various controls
– Ratings
– Objective indicators
(only for the controls)
Cell (1974)
Various heads of state
How do leaders with
different outstanding
leadership styles (i.e.,
charismatic, ideological,
or pragmatic) and
orientations (i.e.,
personalized or
socialized) differ in
Machiavellianism?
Do situational factors
influence the charisma
of heads of state?
– MANOVA
– ANOVA
– Interrater agreement
– Correlation
– Fisher's R to Z
transformation
– Validity
– Reliability
– Correlation
– Regression
– Biographies
– Interviews from
country nationals
– Charisma
– National social crisis
– Nationalism plus
nationalistic movements
– Disrupted youth
– Denial of access
– Prepower following
– Judgments
–
–
–
–
–
DeChurch et al. (2011)
Leadership during
natural disaster
responses
In what ways are the
effects of leadership
important in mission
critical multiteam
environments (e.g.,
responses to natural
disasters)?
Certain situational
factors, namely
nationalism plus
nationalistic movements,
denial of access, and
disrupted youth, are
collectively strong
indicators of a leaders'
charisma.
Functional leaders
positively influence
several aspects
associated with
multiteam processes,
emergent states, and
performance.
– Critical incidents
– Leadership–relevant
incidents
– Leadership strategy
– Leadership
coordinating
– Ratings
– Judgments (regarding
incidents)
– Interrater agreement
Deluga (1997)
U.S. presidents
How does narcissism
relate to charismatic
leadership and rated
Narcissism was
largely positively
related to presidential
– Center for Army's
Lessons Learned
– Defense Technical
Information Center
archives
– Various newspapers
and books
– Various documents
available through
multiple types of sources
(e.g., blogs and
communities of practice)
– Profiles developed
by Simonton (1986a,
1988a) that were
derived from
– Narcissism
– Presidential charismatic
leadership
– Ratings
– Rankings
– Interrater agreement
– Regression
Correlation
Regression
Partial correlation
Path analysis
Validity (scale)
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
Reference
Reference
Leader(s)/type of leader(s)
Research question
Conclusion
Data source(s)
Main constructs of
assessment
Construct codings
performance in U.S.
presidents?
charismatic leadership
and rated performance.
biographical sources
and presidential fact
books
– DeGregio's (1991)
presidential reference
work
– Same sources as
used in Deluga (1997)
– Rated presidential
performance
U.S. presidents
How does proactivity
relate to charismatic
leadership and rated
performance in U.S.
presidents?
Presidential proactivity
positively related to
presidential charismatic
leadership and rated
performance.
Deluga (2001)
U.S. presidents
How does a president's
Machiavellianism relate
to charismatic
leadership and rated
performance?
Presidential
Machiavellianism was
positively related to
charismatic leadership
and rated performance.
– Profiles developed
by Simonton (1986a,
1988a) and Deluga
(1997, 1998)
– Previous studies
Emrich, Brower, Feldman,
and Garland (2001)
U.S. presidents
How does the use of
images in presidential
speeches influence
perceptions of charisma
and greatness?
Presidents who used
more image-based rhetoric were perceived as
being greater and more
charismatic.
– Presidential inaugural
addresses
– Pivotal presidential
speeches
– Various sources
Etheredge (1978)
Various U.S. politicians
Does personality relate
to a leaders' foreign
policy decisions?
– Scholarly works
– Insiders' accounts
– Biographies
– Autobiographies
Eubanks et al. (2010)
Outstanding political
leaders of the 20th
century
With regard to
criticism, how do leader
response strategies
influence follower
reactions and
resolution of the
criticism?
Fiol, Harris, and House
(1999)
U.S. presidents
How do charismatic
leaders generate social
change?
Certain personality
traits (e.g. introversion/
extroversion) seem to
relate to how an
American leader
regards foreign policy
issues.
Collaborative and
confrontational response
strategies seem to be
most effective for
gathering follower
support and continuing
with an agenda, whereas
response strategies that
are avoidant, divert
attention, or attempt to
be persuasive are less
effective.
Charismatic leaders
seem to use consistent
communication
strategies for
– Proactivity
– Presidential charismatic
leadership
– Rated presidential
performance
– Various control
variables
– Machiavellianism
– Presidential charismatic
leadership
– Presidential rated
performance
– Control variables
(friendliness, need for
affiliation, proactivity,
and need for power)
– Image-based rhetoric
– Concept-based rhetoric
– Public charisma and
creativity (based on
Simonton, 1988a)
– Greatness (based on a
previous study)
– Dominance over
subordinates
– Extroversion
– Policy disagreements
– Personality
disagreements
– Ratings
– Rankings
– Objective indicators
– Interrater agreement
– Regression
– Ratings
– Interrater agreement
– Regression
– Ratings
– Judgments
– Correlation
– Regression
– ANOVA
– Ratings
– Judgments
– Correlation
– Counts and
percentages
– Highly structured
academic biographies
– Pathways to
Outstanding Leadership
– Hostile criticism
– Leader appraisal of
criticism
– Leader response
strategy
– Others' reactions to
leader's response
– Resolution of criticism
– Various controls
– Ratings
– Objective indicators
(only for controls)
– Interrater agreement
– Correlation
–Regression
– Exploratory factor
analyses
– Various presidential
speeches (mostly
inaugural addresses or
addresses to congress)
– Communication acts
– Charisma
– Judgments
– Interrater reliability
– ANOVA
– Least squares means
test
1109
(continued on next page)
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
Deluga (1998)
Statistical analyses
1110
Table 2 (continued)
Reference
Leader(s)/type of leader(s)
Research question
Conclusion
generating social
change.
Coach succession,
and at varying times
(e.g., during season and
off-season), seems to
influence team performance more than owner
succession.
NBA coaches and team
owners
Does coach succession
influence team
performance differently
than owner succession?
Holmes and Elder (1989)
U.S. presidents
Which indicators
differentiate the best
and worst U.S.
presidents?
House, Spangler, and
Woycke (1991)
U.S. presidents
Does a president's
personality and
charisma influence his
effectiveness?
Hunter et al. (2011)
Outstanding college
and NFL coaches
Kenney and Rice (1988)
U.S. presidents
Do the leadership styles
of charismatic,
ideological, or
pragmatic leadership
influence how coaches
lead their teams?
Which contextual
factors best predict
presidential greatness?
There is evidence that a
coach's leadership style
affects how that coach
leads the team and
guides the team
members.
A variety of contextual
factors (e.g., whether
the president is
assassinated, served
during wartime)
significantly predict
presidential greatness.
Kynerd (1971)
U.S. presidents
How do five
presidential rankings
compare to one
another?
Of the five sets of
rankings discussed,
they do not seem to be
statistically different
from one another, but
rankings may be
Various indicators
(e.g., publishing books
before assuming office,
being more assertive on
foreign policy issues)
significantly differentiate
the best and worst U.S.
presidents.
A president's
personality and
charisma do indeed
influence presidential
performance, although
different traits are
stronger indicators.
Main constructs of
assessment
Construct codings
Statistical analyses
Unidentified
– Coach succession
– Owner succession
– Team performance
(winning percentage)
– Controls (prior
performance, coaching
ability, coaching
experience)
– Countless indicators
– Objective indicators
– Correlation
– Time series modeling
– Regression
– Chi-square test for
equality
– Ratings, judgments,
and objective indicators
from previous studies
– t-test
– Fisher's exact test for
independence
– Institutional age
– Motives
– Activity inhibition
– Crises
– Editorial charisma
– Behavioral charisma
– Presidential
performance
– Ratings
– Judgments
– Objective indicators
– Factor analysis
– Discriminant validity
– Regression
– Partial least squares
technique
– Leadership type/style
– Leadership orientation
– Various traits
– Various controls
– Ratings
– Judgments
– Objective indicators
(only for controls)
– Interrater agreement
– Correlation
– MANCOVA
– Discriminant function
analysis
– Presidential greatness
– Popular vote victories
– Term(s) served
– Assassination
– War
– Economy
– Success or failure of
passing legislature
– Scandal
– Rankings of presidential
greatness
– Rankings
– Regression
– Rankings (based on
previous studies)
– Correlation
– z-score
– A variety of sources
related to presidential
studies
– Data from Winter
(1987)
– Letters and speeches
written by presidents
– Letters and speeches
in autobiographies
– Memoirs and diaries
written by presidents
– Biographies
– Editorials from New
York Times
– Historical biographies
– Chicago Tribune
– Various sources
– Previous studies that
ranked presidents on
greatness
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
Giambatista (2004)
Data source(s)
Reference
Leader(s)/type of leader(s)
Research question
Conclusion
Data source(s)
influenced by raters'
political bias.
Certain life events are
better indicators of
particular leader types
or orientation than
others.
Outstanding leaders of
the 20th century
Do certain
developmental events
differentially predict
leadership type and
orientation?
Lilienfeld et al. (2012)
U.S. presidents
How do certain
psychopathic
personality traits relate
to the U.S. presidency?
Marmor-Lavie and
Weimann (2005)
Israeli politicians
Can a reliable and valid
measure be developed
regarding the use of
emotional appeals in
modern political
campaigns?
McCann (1992)
U.S. presidents
Which formula best
predicts presidential
greatness?
McCann (2005)
Deceased U.S. presidents
How does the
relationship between
conscientiousness and
death age apply to U.S.
presidents?
Boldness, via fearless
dominance, positively
related to several
factors of presidency
(e.g., performance,
persuasiveness),
whereas impulsive
antisociality was
related to factors of
negative job
performance.
A reliable and valid
measure was
developed, with results
suggesting that
emotional appeals
differ depending on
certain contextual
factors, but also the
party that a politician is
affiliated with.
A 6-variable formula
(including 5
personological variables and one zeitgeist
variable) predicted a
large amount of variance (91%) in presidential greatness.
Conscientiousness
positively correlated
with death age and
exercise, and negatively
correlated with
smoking and drinking.
The other Big Five
factors were not
correlated with death
age, exercise, smoking,
or drinking.
Construct codings
Statistical analyses
– General history texts
– Biographical websites
– Academic biographies
– Originating event
– Turning point event
– Anchoring event
– Analogous event
– Redemption event
– Contaminating event
– Various thematic
constructs
– Ratings
– Judgments
– Objective indicators
(only for controls)
– Rater expertise
– Previous studies
about U.S. presidents
(e.g., Simonton, 1986a)
– Polls
– Countless indicators
– Ratings
– Objective indicators
– Televised political
spots
– Emotional arousal (for
various emotions)
– Emotional appeal to
hope, pride, sympathy,
warm-heartedness,
anger, and fear
– Ratings
– Objective indicators
– Mean
– Median
– Mode
– Validity
– Interrater reliability
– t-test
– MANOVA
– Previous research on
presidential greatness
– Presidential greatness
– Various personological
dimensions
– Various situational
variables
– Zeitgeist
– Ratings (from
previous research)
– Objective indicators
– Correlation
– Regression
– Big Five ratings
– Health practice
assessments
– Death age
– Inauguration age
– Birth year
– Big Five personality
variables
– Health practices
– Ratings
– Judgments
– Objective indicators
– Correlation
– Regression
– Logistic regression
– Kappa agreement
– Q-sort
– Interrater agreement
– Chi-square frequency
analysis
– Correlation
– MANCOVA
– Discriminant function
analysis
– Regression
– Interrater reliability
– Correlation
– General linear
modeling
– Incremental validity
1111
(continued on next page)
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
Ligon et al. (2008)
Main constructs of
assessment
1112
Table 2 (continued)
Leader(s)/type of leader(s)
Research question
Conclusion
Data source(s)
Main constructs of
assessment
Construct codings
Statistical analyses
Miller and Stiles (1986)
U.S. presidential
candidates and U.S.
presidents
Familiarity has increased
from 1920 to 1981, and
familiarity also increases
from acceptance speech
to inaugural address.
– Coding per verbal
response mode
– Intercoder reliability
– t-test
– Correlation
Outstanding violent
and non-violent
leaders of the 20th
century
– Transcripts of U.S.
presidential
nomination acceptance
speeches and inaugural
addresses from a
variety of sources
– Preliminary sources
included historical
summaries,
biographical listings,
and web sites
– Biographies
– Familiarity
Mumford et al. (2007)
Does level of verbal
familiarity differ across
time with respect to
presidential nomination
acceptance speeches
and inaugural addresses?
Are ideological leaders
more likely to incite
violence than
non-ideological
leaders?
– Ratings
– Judgments
– Objective indicators
– Interrater agreement
– Correlation
– Regression
– Discriminant analysis
Mumford (2006)
Charismatic, ideological,
and pragmatic leaders
What theoretical
dimensions
differentiate
charismatic, ideological,
and pragmatic leaders?
Charismatic,
ideological, and
pragmatic leaders differ
across a variety of
theoretical dimensions.
– Histories
– Biographies
– Encyclopedias
– Ratings
– Objective indicators
– Q-sort
– Discriminant analysis
– Regression
Murray (1928)
Great orators (including
religious leaders)
Can traits present in
early life, especially
genius, influence
whether one will
become a great orator?
– The Early Mental
Traits of Three Hundred
Geniuses
– Ratings
– N/A
O'Connor, Mumford,
Clifton, Gessner,
and Connelly
(1995)
Charismatic leaders
What personality
characteristics best
differentiate between
personalized and
socialized charismatic
leaders with regard to
their destructiveness?
Traits early in life,
particularly genius and
perseverance, can
influence one's status
later in life as a great
orator. Those traits
should be promoted
in schools so as to
create great orators.
The studied constructs
of interest seem to
influence the
amount of
destructiveness a
charismatic leader
engages in.
– Leadership type
– Violence/non-violence
criteria
– Performance criteria
– Several predictor
variables at the
individual, group,
organizational, and
environmental levels
– Various controls
– Problem solving
– Communication
– LMX relationships
– Developmental influences
– Political tactics
– Performance
– Various personality
traits
– IQ
– General history texts
– Almanacs
– Biographical listings
– Encyclopedias
– Historical, scholarly
biographies
– Outcome uncertainty
– Need for power
– Object beliefs
– Negative life themes
– Narcissism
– Fear
– Self-regulation
– Ratings
Panagopoulos (2004)
U.S. congressional
candidates
Gender does influence
campaign
communications
strategies, but not in
very sizeable ways.
– Database of political
advertising collected by
Campaign Media
Analysis Group
– Tone of ad
– Style of ad
– Policy content of ad
– Various factors
– Judgments
Rubenzer, Faschingbauer,
and Ones (2000)
U.S. presidents
Are different campaign
communications
strategies used
between male and
female congressional
candidates?
Can standardized
personality tests
be used in historical
research?
– Interrater agreement
– Correlation
– MANOVA discriminate
function analysis
– t-test
– Cross-validation
– ANOVA
– Causal analysis
– Means
– t-test
– Multivariate probit
analysis
By using the NEO
PI-R and presidential
experts, results
support the notion
– Rater expertise
(raters were
presidential experts)
– Big Five personality
factors
– Positive and negative
valence
– Ratings
The attributes associated
with ideological
leadership influenced the
amount and occurrence
of institutional and
cultural violence.
– Modified interrater
reliability
– Q-sort (including Q
analysis and
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
Reference
Reference
Leader(s)/type of leader(s)
Research question
Conclusion
Main constructs of
assessment
Construct codings
that standardized can
be reliably used in psychohistorical research.
– Previous studies of
presidential greatness
– Job performance
– Intelligence factors
– Behavior as president
– Other variables
– Presidential success
– Eminence
– Father's status
– Intelligence
– Education
– Major vocation
– Life span
– Vocational choice
– Birth year
– Military success
– Casualties
– Various individual
factors
– Various situational
factors
– Various interaction
terms
– Leadership ratings
(based on a previous
study)
– Transition into
presidency
– Various administration
events
– Various biographical
predictors
– Intelligence
– Morality
– Leadership
– Eminence
– Life span
– Reign span
Simonton (1976)
Eminent creators and
leaders
Were the analyses
conducted by Cox
(1926) susceptible
to certain artifacts?
The analyses conducted
by Cox (1926) might
have been influenced
by data reliability and a
time-wise sampling
bias.
– Data gathered and
analyzed by Cox (1926)
Simonton (1980)
Military leaders of land
battles
Which factors
predict military
success during land
battles?
A military leader's years
of experience, winning
streaks, willingness to
take the offensive, and
divided command are
significant determinants
of a victory.
– Military encyclopedia
– Military dictionaries
Simonton (1981)
U.S. presidents
Various determinants
and biographical
variables influence
presidential
performance and
greatness.
– Several archival
sources of U.S.
presidents
Simonton (1983)
Hereditary monarchs
of European nations
(circa 900–1800 A.D.)
What are the principal
determinants of the
rated greatness of
American presidents,
and can presidential
performance be
predicted using
preelection
biographical variables?
Which
intergenerational
factors influence
monarchs the most,
as well as which
individual differences
are affected by those
factors?
– General reference
works
– Biographical
dictionaries
– Historical
encyclopedias
– National histories
Simonton (1984)
European hereditary
monarchs
To what degree do
situational factors,
as opposed to
individual factors,
influence ruler
eminence?
Simonton (1985)
Political leaders (U.S.
presidents, vice
presidents, and
congressmen)
Is the vice-presidential
effect influenced more
by individual factors or
situational factors?
Intergenerational
transfer of traits
is influenced by
genetic, rolemodeling, cohort,
and sociocultural
effects, suggesting
that no one process
is necessarily more
impactful than another.
Although leader
eminence is strongly
influenced by the
number of significant
events during leadership,
individual factors still
play a large role.
The vice-presidential
effect seems to be
more strongly
influenced by
situational factors.
Statistical analyses
Q-factor analysis)
– Averages and standard
deviations
– Correlations
– Judgments
– Objective indicators
– Correlation
– Regression
– Judgments
– Objective indicators
– Correlation
– Regression
– Discriminant analysis
– Ratings
– Judgments
– Objective indicators
– Correlation
– Regression
– Ratings
– Objective indicators
– Correlation
– Regression
– Factor Analysis
– Same constructs as
Simonton (1983)
– Significant events
– Ratings
– Objective indicators
– Correlation
– Partial correlation
– Regression
– Various reference
works
– Almanacs
– Statistical books
– Presidential Vetoes,
1789–1971
– Countless variables
– Judgments
– Objective indicators
– Correlation
– Discriminant analysis
– Time-series analysis
(continued on next page)
1113
– Same sources as
Simonton (1983)
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
Data source(s)
Leader(s)/type of leader(s)
Research question
Conclusion
Data source(s)
Main constructs of
assessment
Construct codings
Statistical analyses
Simonton (1986a)
U.S. presidents
How do presidents vary
in personality based on
the Gough Adjective
Check List?
Intellectual Brilliance
was the only factor
that significantly
predicted presidential
greatness.
– Personality attributes
(per the Gough
Adjective Check List)
– Ratings
– Objective indicators
– Correlation
– Reliability
– Factor analysis
– Cluster analysis
Simonton (1986b)
U.S. presidents
When given limited
information about
leaders (e.g., U.S.
presidents), do naïve
raters differ markedly
from historians?
– Semantic components
of leader schema
(strength, activity,
and goodness)
– Greatness
– Various historiometric
predictors
– Judgment
– Ratings
– Objective indicators
– Interrater reliability
– Correlation
– t-test
– Regression
Simonton (1987)
U.S. presidents
How does a president's
inflexibility interact
with situational factors
to influence veto
behavior?
– Presidential Vetoes,
1789–1971
– Previous research
– Variables about vetoes
– Political variables
– Inflexibility
– Dogmatism
– Ratings
– Judgments
– Objective indicators
– Correlation
– Time-series regression
analysis
Simonton (1988a)
U.S. presidents
Can the presidents be
differentiated based on
attributes derived from
biological descriptors,
do those attributes
belong to factors, and
do those factors relate
to other variables?
– Biographical sources
– Several previous
studies
– Presidential style
– Various personality
traits, biographical
experiences, and
performance criteria
– Ratings
– Objective indicators
– Correlation
– Regression
– Alpha reliability
– Factor analysis
– Cluster analysis
Simonton (1988c)
Eminent creators,
leaders, and celebrities
from Chinese civilization
(circa 840 B.C. to
1979 A.D.)
Do major and minor
historical figures
co-fluctuate across
time, as well whether
intra-field generational
autoregression occurs
regarding
achievements?
– Histories
– Anthologies
– Biographical
dictionaries
– Time series
(20 year intervals)
– Categories of
achievement
– Major versus minor
figures
– Objective indicators
– Correlation
– Time series/trend
analysis (via regression)
Simonton (1992)
Eminent creators and
leaders of Japan (circa
580–1939 A.D.)
Which factors influence
the number of
distinguished women
over time in
male-dominated
cultures?
Attributions of
greatness to leaders
do no significantly
differ between naïve
raters and historians
when given limited
information about
those leaders.
Whether a president
vetoes can be the
result of an interaction
between his
inflexibility and certain
situational factors
(such as electoral
mandate and which
party controls
Congress).
The presidents were
reliably discriminated
on many items
concerning presidential
style, and those items
collapsed into five
dimensions. Likewise,
those dimensions
related to other
variables of interest.
Major and minor
historical figures
co-fluctuate across
time, and the achievements of previous
generations can
predict the achievements of subsequent
generations.
Female literary and
nonliterary eminence
changed along with
male literary activity,
power and aggressive
behavior, and ideology
– Various biographical
reference works
– Presidential fact
books
– American history
compendia
– Data and information
from Simonton's
previous studies on
presidential greatness
– Japan Biographical
Dictionary & Who's
Who (1965)
– Kodansha
Encyclopedia of
Japan (1983)
– Time series
(20 year intervals)
– Female achievement
– Male literary creativity
– Male machismo
activities
– Male ideologies
– Objective indicators
– Alpha reliability
– Correlation
– Time series analysis
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
Reference
1114
Table 2 (continued)
Reference
Leader(s)/type of leader(s)
Research question
Conclusion
Eminent creators and
leaders of Japan (circa
580–1939 A.D.)
Do foreign ideas and
peoples affect a nation's
achievements? Is there
much of a time gap
between cultural
acceptance and
achievement influence?
Which domains of
achievement are most
influenced?
Simonton (2001)
U.S. presidents
Do certain variables
adequately predict
rankings of presidential
performance?
Simonton (2006)
U.S. presidents
How does estimated IQ
scores of U.S. presidents
relate to their
performance?
Simonton (2008)
Eminent African
Americans
Spangler and House (1991)
U.S. presidents
Because previous
studies involving the
relationship between
giftedness and genius
have almost solely
focused on
majority-culture samples, how does that relationship apply to
minority-culture
samples?
How is presidential
performance influenced
by individual motives?
Strange and Mumford
(2002)
Charismatic and
ideological leaders of the
20th century
Do different mental
models influence
different vision-based
leadership styles?
Foreign influences
typically had positive
effects on most
domains of creative
achievement, and
leadership domains
were positively
associated with
national openness to
those foreign
influences.
A 6-variable equation
(including years in office, war years, scandal,
assassination, heroism
in war, and intellectual
brilliance) successfully
predicted rankings of
presidential
performance.
Estimated IQ scores
correlate with evaluations of presidential
leadership
performance.
Both adult eminence
and creative
achievement positively
correlated with early
giftedness in a
minority-culture sample (specifically, African
Americans).
The number of nots that
a president used in
speeches and letters
measured a style of
using power.
Construct codings
Statistical analyses
– Time series (20 year
intervals)
– Domains of
achievement
– External influences
(outside influences,
travel abroad, and
eminent immigrants)
– Objective indicators
– Correlation
– Cross-correlation
– Reference works
– Previous studies by
Simonton
– Performance rankings
– Years in office
– War years
– Assassination
– Scandal
– War hero
– Intellectual brilliance
– Ratings
– Objective indicators
– Correlation (including
validity)
– Alpha reliability
– Regression
– Previous studies and
various sources (in
particular: Cox, 1926;
Rubenzer et al., 2000;
Simonton, 1986a)
– The Encyclopedia of
Black America
– The African-American
Almanac
– The Encyclopedia of
African-American Culture and History
– Various majority- and
minority-culture
sources
– IQ
– Intellectual brilliance
– Openness to experience
– Leadership
performance
– Archival eminence
– Creative achievement
– Giftedness
– Birth year
– Living contemporary
– Gender
– Achievement domain
– Ratings
– Objective indicators
– Missing-values estimation methods
– Correlation
– Ratings
– Objective indicators
– Regression
– Motive scores from
Winter (1987)
– Letters and speeches
from collections,
autobiographies,
memoirs, and diaries
– Biographies
– Preliminary sources
included general
history texts and
biographical websites
– Biographies
– Motive
– Activity inhibition
– Presidential
performance
– Ratings
– Objective indicators
– Correlation
– Regression
– Charismatic behaviors
– Ideological behaviors
– Leader performance
– Leader contributions
– Various controls
– Ratings
– Objective indicators
– Correlation
– Regression
– MANOVA
– Discriminant analysis
– Biographical
dictionaries
– Same sources as
Simonton (1992)
(continued on next page)
1115
Vision-based leadership
styles differ: ideological
leaders stress personal
values and standards
whereas charismatic
leaders adhere more to
social needs and change
requirements.
Main constructs of
assessment
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
Simonton (1997)
Data source(s)
1116
Table 2 (continued)
Leader(s)/type of leader(s)
Research question
Conclusion
Data source(s)
Main constructs of
assessment
Construct codings
Statistical analyses
Suedfeld and Bluck (1988)
Political leaders of
countries that engaged in,
or were recipients of,
surprise attacks
Do leaders' integrative
complexities change
before a surprise attack
occurs?
– Ratings
– Interrater reliability
– Integrative complexity
– ANOVA
– t-tests
Revolutionary leaders
How does a
revolutionary leader's
conceptual complexity
relate to his success?
– Proceedings of UN
General Assembly or
Security Council
– Speeches
– News conferences
– Various sources (e.g.,
letters to colleagues,
theoretical treatises,
defense pleas)
– Integrative complexity
Suedfeld and Rank (1976)
–
–
–
–
– Judgments
– Correlation
– ANOVA
Suedfeld, Tetlock, and
Ramirez (1977)
Political leaders of
countries involved in the
Middle East conflict
Do leaders' integrative
complexities change
before an outbreak of
war?
– Ratings
– Interrater reliability
– Integrative complexity
– Correlation
– ANOVA
U.S. senators in the 82nd
Congress
How does a senator's
integrative complexity
relate to his or her
personality and
standing on foreign
policy (i.e. isolationist,
ambivalent isolationist,
or nonisolationist)?
– Speeches made in UN
General Assembly and
Security Council in the
months prior to outbreaks of major
hostilities
– Congressional
speeches relevant to
foreign policy
– Integrative complexity
Tetlock (1981)
– Integrative complexity
– Evaluative assertion
– Judgments
– Ratings
– Interrater reliability
– Integrative complexity
– Correlation
– ANOVA
– Discriminant analysis
Tetlock (1983)
U.S. senators
Does a senator's
political ideology relate
to his or her integrative
complexity?
From the attackers'
perspective, a decrease
in complexity occurs as
the time of a strategic
surprise attack nears.
Revolutionary leaders
who were more
adaptable to the
environmental
demands they
encountered were more
successful.
Complexity decreases
the nearer an outbreak
of war is, suggesting
that complexity could
predict when hostilities
may erupt.
Foreign policy
statements made by
isolationists were the
least complex, and
typically expressed
more positive attitudes
toward their in-group
and more negative attitudes toward an
out-group.
Conservative senators
typically made less
complex statements
than did moderate and
liberal senators.
– Speeches from the
Congressional Record
– Ratings
– Objective indicators
– Interrater reliability
– Integrative complexity
– ANOVA
– ANCOVA
– Regression
Thoemmes and Conway
(2007)
U.S. presidents
How do U.S. presidents'
integrative complexity
differ?
Presidents' integrative
complexity differ based
on a number of factors.
– State of the Union
speeches
– Integrative complexity
– Political party
– Education
– Age
– Length of service in the
Senate
– Integrative complexity
– Various environmental
measures
– Various personality
measures
– Ratings
– Objective indicators
Thorndike (1936)
Male members of
European royal families
To repeat Woods's work
to determine on
Monarchs accuracy of
assessment regarding
the relationship
between intellectual
ability and estimability
Results replicated the
positive and moderate
relationship between
intellect and morality,
suggesting that more
intelligent leaders may
– Heredity in Royality
– IQ
– Morality
– Ratings
– Interrater reliability
– Integrative complexity
– Correlation
– MANCOVA
– Regression
– ANOVA
– ANCOVA
– Multilevel modeling
– Effect sizes
– Correlation
Polarized contrast
Qualified contrast
Integrative comparison
Conceptual complexity
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
Reference
Reference
Leader(s)/type of leader(s)
Conclusion
Data source(s)
Wallace, Suedfeld, and
Thachuk (1993)
Leaders during the Gulf
Crisis
of character (i.e.,
morality).
Does a leader's
integrative complexity
influence their
behaviors?
Wendt and Muncy (1979)
U.S. vice presidents
Which personality
variables influence the
political success of U.S.
vice presidents?
Winter (1980)
Southern African leaders
Winter (1987)
U.S. presidents
Do certain factors
influence the type of
motive imagery used by
Southern African
leaders?
What are the various
motive profiles of U.S.
presidents according to
three different
leadership models, and
how do those profiles
relate to leader appeal
and performance?
be more morally
upstanding.
Changes in integrative
complexity, especially
during stressful times,
can indicate certain
behaviors.
Certain factors
(activity–charisma,
likability, and shrewd
persistence) influence
U.S. vice presidential
success.
A leader's war
disposition and activity
correlate with the types
of motive imagery used.
Yammarino, Mumford,
Serban, and Shirreffs
(2012)
Political leaders
Zullow and Seligman
(1990)
U.S. presidential
candidates
Which type of leader
(personalized,
socialized, charismatic,
ideological, or
pragmatic) is more
likely to be the victim of
assassination or
attempted
assassination?
Do the ways in which
presidential candidates
frame explanations
about bad events
predict defeat?
Pessimistic ruminators
were much more likely
to lose.
Construct codings
Statistical analyses
– Texts from New York
Times
– Texts from embassies
– Integrative complexity
– Ratings
– Interrater reliability
– Integrative complexity
– Excerpts from
Madmen and Geniuses
– Dozens of
characteristics
– Judgments
– Objective indicators
– Regression
– Factor analysis
– Interjudge variability
– Verbatim responses to
reporters' questions
– Verbatim transcripts
published in weekly
news magazines
– Presidents' first
inaugural address
– Achievement,
affiliation, and power
motive imagery
– War disposition
– Activity
– Achievement
– Affiliation–intimacy
– Power motive imagery
– Ratings
– Judgments
– Split-half reliability
– Correlation
- Validity
– Ratings
– Correlation
– Previous
historiometric studies
that included political
leaders
– Various websites
(including Wikipedia
biographies, university
website biographies,
and historical
biographies)
– Nomination
acceptance speeches
(obtained from
published proceedings
or the New York Times)
– Campaign itineraries
– Whether leader was
socialized or personalized
– Whether leader was
charismatic, ideological,
or pragmatic
– Victim of assassination
– Target of assassination
attempts
– Various controls
– Ratings (from
previous studies)
– Objective indicators
– Chi-square analysis
– Log linear models
– Logit/logistic regression
– Discriminant analysis
– Causal explanations for
bad events
– Pessimism
– Rumination
– Explanatory style
– Vote spread
– Change in spread
– Ratings
– Interrater agreement
– Z scores
– Correlation
(zero-order and partial)
Q11
1117
Reference: citation.
Leader(s)/type of leader(s): the leaders or type of leaders being assessed.
Research question: the broad research question being studied.
Conclusion: the general conclusion of the study.
Data source(s): sources from which data were gathered.
Main constructs of assessment: the constructs the leaders were assessed on.
Construct codings: how the constructs were coded.
Statistical analyses: types of statistical analyses used.
A president's greatness
in part is associated
with his motives, and
likewise a higher match
between a president's
motive profile and
contemporary society
resulted in higher
presidential appeal.
Socialized charismatic
leaders were most
likely to be victims of
assassination or
attempted
assassination, whereas
socialized pragmatics
were least likely to be
victims.
Main constructs of
assessment
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
Research question
1118
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
beneficial about the study of outstanding leadership, then, is that by virtue of the type of information available about them, we are
able to discern contextual details that would not be afforded if the unit of analysis (i.e., non-outstanding leader) were not under
such academic scrutiny. In short, because these individuals are considered “outstanding” in terms of influence, there are copious
open-source documents detailing observable, and more importantly quantifiable, differences among them in regard to behaviors,
developmental influences, and performance.
Given the above examples of what historiometric approaches can provide, it is surprising that we do not see more of these studies
published in Leadership Quarterly to test develop, and refine leadership theories. Like all empirical methods, this approach does suffer
some limitations, and these disadvantages should be assessed carefully in relation to the advantages afforded. In addition, because
few graduate programs offer training in this approach, its underutilization in our field may also be due to lack of clarity around
methods required for its execution. Thus, in the remaining sections we will provide an examination of methodological best practices
for leadership historiometry, followed by a balanced analysis of the strengths and weaknesses of this technique.
4. Methodological best practices
To date, page limits have restricted leadership scholars from fully providing step-by-step details on the methodological rigor
required to some degree. While other broader social science outlets have published methodological pieces on variants of this
technique (for example, see Simonton, 2010) in relation to other domains, when examining patterns of individual differences in
leadership, a particular set of procedures must be employed for at least two reasons.
First, because historical data about outstanding leaders is quite substantial and heterogeneous in nature, a content-driven
coding scheme must be a priori defined in relation to the theories and sample at hand. While application of standard individual
difference assessments has been applied successfully in some studies (Simonton, 1986a,b), we have found particular success in
generating a theoretically-driven, psychometrically sound list of behavioral items hypothesized to differ among leaders (Hunter
et al., 2011; Ligon et al., 2008; Mumford, 2006). These items, anchored by clear observational markers (e.g., type of University
attended), are reliably identifiable in the factual, historical writing style found in most of the documents used in historical
analysis. Second, because the subjects under examination are at times well-known and sometimes elicit rater biases (e.g., halo,
Simonton, 1990) it is occasionally important to remove identifiers from the data in addition to employ a coding method that
scrambles the observations across leaders and across raters. For this reason, we recommend using a multi-step coding technique,
where observations are sorted into categories and/or assigned ratings by several iterations and across independent rater teams
(McKeown & Thomas, 1988; Mumford, Gaddis, Strange, et al., 2006).
In short, application of historiometry to the study of leadership requires six inter-related steps: 1) model specification, 2)
sampling plan formation, 3) content coding scheme development, 4) materials preparation, 5) coding logistics, and 6) descriptive
and multivariate analysis. Table 2 provides an overview of exemplar studies using variants of these techniques to examine
leadership in particular.
4.1. Model specification
Similar to all empirical examinations of leadership, the initial step in this methodology revolves around specifying
hypothesized differences between leaders in the population of interest. Because this technique requires a fair amount of
historically verifiable secondary-source data, research questions that allow this technique must seek to compare exemplars of
similar historical notoriety and timeframe on constructs of interest. Thus, there are three typical types of theoretical models
employed in the typical historiometric study: 1) descriptive studies seeking commonalities among leaders, 2) predictive studies
assessing differential relationships to performance criteria, and 3) combination of the two for theory development. When the goal
is to describe commonalities in a sample of outstanding leaders, researchers ask the question: what repeatable patterns can we
see in this group? In a hallmark study representing this type of model specification, Murray (1928) examined a sample of
individuals deemed “Great Orators” to determine if a common set of traits in early life were related to later status as an
outstanding orator. A goal in this study was to determine what kind of educational activities should be promoted in early
education to influence subsequent oration skills. As a result, activities under model specification required copious rating scales to
be developed measuring artifacts of “genius” as described in historical sources about the individuals in question.
While Murray's work exemplified early efforts to describe commonalities, the second type of theoretical model deployed in
historiometry is intended to differentiate sample members based on some common criterion. Simonton's (1988b, 2001) work to
differentiate U.S. Presidents in terms of rating and ranking performance data, respectively, represents a model intended to capture
performance on a continuum, while Yammarino and colleagues' (2012) recent examination of predictors of assassination
dichotomize the performance criterion (i.e., assassinated versus not). Each of these types of studies is intended to assess
leadership characteristics related to some subsequent outcome (i.e., performance, assassination) and they provide some evidence
as to the strength, direction, and robustness of these relationships. For example, Simonton (2001) examined contextual
characteristics such as degree of scandal, war, and conflict during terms as they related to subsequent presidential rankings by
subject matter experts (SMEs). Yammarino and colleagues (2012) examined a priori classified leadership style, based on the CIP
model, of assassinated leaders when compared to a matched sample (heads of state and world leaders) of non-assassinated
leaders in an effort to determine which leadership style might be most vulnerable to assassination.
Finally, the third model used in historiometry, rooted in theory development, combines the first two types of research
questions above to provide convergent and divergent validity evidence about the inferences that can be made about styles of
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
1119
leadership. To date, Mumford (2006) is the most comprehensive effort in this vein that relies upon a historiometric approach. He
executed a series of studies to investigate the homogeneity within the leader styles of charismatic, ideological, and pragmatic
leaders (CIP) in terms of developmental influences, problem solving, communication, political tactics, and follower interactions
(Mumford, Scott, et al., 2006). In addition, he examined the relationship of these patterns of differential characteristics between
each style with performance criteria such as sustainability, valence (i.e., positive or negative), and scope of influence (Mumford,
Strange, et al., 2006). This set of research questions required careful sampling to examine exemplars of each of the leader styles, as
well as a content coding scheme used to discriminate them on hypothesized theoretical differences and performance criteria.
4.2. Sampling plan formation
How the population of outstanding leaders is sampled depends on which of the three general research questions described
above are posed. For example, in Murray (1928). As a great deal of Simonton (1988b, 2001, 2008). When the research question
seeks to differentiate leaders on a temporal or situational bound set of criteria, however, more targeted sampling must be
employed. For example, in DeChurch et al. (2011) study of leaders during natural disasters, leaders were sampled through the
Army and the Defense Technical Information Center's archives in order to examine leader behaviors such as coordination, strategy
formation, and communication. In a study examining the relationship between conscientiousness and death age, McCann's
(2005) sample included only deceased U.S. Presidents.
The nature of the research question dictates the desired characteristics of the sample members, and thus sampled populations of
outstanding leaders will vary across studies accordingly. However, there are at least four best practices associated with the overall
sample plan formation across most historiometric studies: 1) sample membership validation, 2) time period alignment, 3) selecting
quality source data, and 4) selecting multiple leaders and types of source data. First, in some cases sample membership is clear;
Yammarino and colleagues (2012) sought to examine the leadership style of every leader assassinated in the 20th Century, which
helped make sample inclusion a dichotomous, clear cut decision. However, in other cases, particularly when research questions are
geared around validating a particular stylistic framework, a priori assignment to such categorical distinctions is more complicated.
This can be done via reliance upon previous classification schemes (Eubanks et al., 2010), empirical clustering (Strange & Mumford,
2002), or subject matter expert (SME) ratings of category assignments (Ligon et al., 2010). For example, as Strange and Mumford's
(2002) study of ideological leadership was the first that sought to distinguish this style from a charismatic style, a cluster analysis
(Aldenderfer & Blashfield, 1984) was first applied to identify what behaviors (e.g., focus on the past in messages — ideological)
clustered together empirically. Next, performing a discriminant function analysis (Klecka, 1980), each leader was then assigned a
score on these clusters to identify who manifested higher indices of one leadership style or the other. Leaders were then classified as
ideological (high scores on ideological functions), charismatic (high scores on charismatic functions), or mixed (high scores on both
functions) based on these empirically derived distinctions. In subsequent studies examining these theories (Mumford, 2006),
members were sampled who evidenced strong (i.e., high scores on one or the other type of function) ideological or charismatic leader
styles.
While empirically derived techniques allow for classification of leaders based on clusters of behaviors, this step requires a great
deal of resources to execute and is not always the best strategy given the research question at hand. Thus, the reliance of SME ratings
to validate leader classifications is often employed in historiometric sampling plans. For example, in a study examining non-violent
ideological leaders in regions and organizations relatively comparable to violent ideological leaders, a SME group of religious,
geographic, and social movement scholars was used to help identify populations of interest and validate our selected sample
assignments (Ligon et al., 2010). Because these area-specific SMEs are typically not also leadership scholars per se, the process of
developing carefully constructed scales is critical for valid ratings of leaders into classified groups. To do this, first an initial
classification, based on a sincere effort from the study Principal Investigator (PI) and research team in reviewing material for potential
leaders to include in the sample, is assigned. Next, evaluations of the validity of these assignments to cells/types are completed via
SME ratings on a series of Likert scales about classification scheme. Fig. 1 shows the rating scales we provided for SMEs in making
assignments to some categories of interest: leaders of non-violent, ideological organizations whose height of power was during the
20th Century (Ligon et al., 2010).
As noted in Fig. 1, another component of the sampling plan is to identify the period of interest from which to sample behaviors
of the leader. For study-specific questions about time periods, this is defined by the research question. For example, in Simonton's
(2008) study of eminent African Americans, he sampled activities during leaders' early (childhood and education timeframe)
developmental periods to examine relationships between formative events and subsequent performance. However, when the
research question assesses behaviors related to productive leadership activities (e.g., speeches, leader–member exchange
relationships), one approach is to a priori identify the leader's height of power and sample such events during this timeframe.
From a developmental standpoint, this is likely when the leaders' most prototypic mode of executing will manifest; events
sampled in early career might not afford a full examination of the leader's style that would later dictate performance. In addition,
at the end of power, the situational conditions might not afford full expression of leadership style as control gradually passes to
others. From a pragmatic standpoint, we have also found that this is when leaders are “watched” the most; studying leaders
during the height of power may yield the most information about his/her leadership style and contextual influences associated
with it as most academic biographers and journalists will record events from this time frame. Given this predilection, it is also
important to note that the model specified should dictate this time period of sampled behavior. For example, studying leader
decisions leading up to loss of power might also warrant insight into behaviors to avoid or minimize. Whatever within-subject
time period is selected for investigation (e.g., rise to power, peak of power), however, it is also important that other time specific
1120
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
boundary conditions be considered in accordance with designated temporal reference. For example, in Hunter et al.'s (2011)
study of college and NFL football coaches, the aim was to investigate only those coaches who were achieved success at the
pinnacle of their profession. Thus, although football was clearly played (and coached) prior to the instantiation of formal
championships, only coaches who won national collegiate championships or Superbowls were included in the final study sample.
The key takeaway here is that given the complexity of data collection associated with these efforts and the ambiguity such
speculation requires, we recommend a priori designating the appropriate within-subject time period as well as time-based
boundary conditions for case inclusion.
The last two practices in sampling for historiometric studies are related: identifying quality and ensuring reliable source data. The
information used to obtain qualitative source material about sample members is quite heterogeneous across studies, but some
general guidelines can be inferred. The common thread between all of the historiometric studies we found was the reliance upon
domain experts — often academics devoting significant portions of their research programs to objective analysis of these leaders — to
assist sample development by using their techniques to gather and report information in a structured, standardized manner. For
example, Simonton (1986a, 1986b, 1988a, 1988b) has developed presidential profiles that others have used for studies examining the
U.S. Presidents (Deluga, 1997, 1998, 2001), while DeGregorio (1991). Although, examples also abound where less academic resources
are used in tandem to these data sources; DeChurch and colleagues (2011) relied upon secondary sources such as blogs and
newspaper reports to cross-reference archived accounts of how national crises were handled. Thus, when the events are highly
publicized in nature, a variety of sources can be used to draw conclusions. Given the variability in source quality, however, statistical
control must be exercised to at least partially account for quality concerns and somewhat standardize the type of information
gathered. Ways to handle source quality variance through application of control variables will be discussed in the next section on
content coding schemes.
Finally, it is imperative that multiple sources, events, and leaders are sampled to assess behavior to draw reliable, generalizable
conclusions about relationships. There are copious high quality studies applied to one leader where researchers obtain quantitative
metrics about a given case, but we argue these studies are not indicative of a true historiometric approach. For example, in Abe's
(2011) study of Alan Greenspan's speeches during economic downturns, conclusions were able to be drawn about that particular
leader's use of psychological distancing, but little could be generalized about leadership in general. Thus, we argue that three defining
hallmarks of historiometric studies are sampling: 1) multiple members of a given population of leaders, 2) multiple types of
behaviors, and 3) multiple sources depicting those leader behaviors. Methodologically related, criterion data are strengthened if they
can be gathered from sources independent of the predictor data, limiting single-source biases that can be introduced. In a study
examining organizational characteristics as they related to terrorist attacks, Asal and Rethemeyer (2008) relied upon the Global
Terrorism Database (National Consortium for the Study of Terrorism and Responses to Terrorism (START), 2012) to gather criterion
data regarding attack lethality, while predictor data was sampled from external news accounts and organizational websites.
4.3. Content coding scheme
As the central goal in historiometric studies is to transform historically verifiable qualitative source information into quantitative
data, this third methodological step requires a particularly stringent set of techniques. Similar to the above section on sampling plan,
model specification drives what this content coding scheme will include, and thus also dictates cross-study differences in variables
assessed. Despite these cross-study differences, there is a set of best practices associated with our approach to leadership
historiometry. Acknowledging that there are other frameworks for content coding (Simonton, 2010), for studies of leadership we
advocate a set of techniques that draw upon psychometric principles with a particular focus on biodata, or life history, item generation
best practices (Mumford, Stokes, & Owens, 1990). Thus, the goal of this section is to provide a description of the general framework
recommended for content coding, followed by a specific discussion of exemplar coding schemes in predictor, criterion, and control
data in these studies.
First, the coding scheme we recommend requires multiple trained coders for each set of constructs (predictors, criteria, and
controls). This is counter to many historiometric studies we reviewed, where approaches largely rely on coding from one individual,
albeit an individual familiar with both the sample and variables under examination (and often the study PI). For example, in a study of
Please review the following list of leaders and his/her inclusive years for height of power. Next to each name,
indicate your rating about each individual’s position (for leaders with whom you are unfamiliar, please write
“unfamiliar”):
1– disagree
2 – agree
3 – strongly agree
1. _____ This is a former leader of non-violent organization.
2. _____ This is a former leader of an ideological organization (organizationdevoted to advancing a certain
belief system rather than maximizing profits; profits are sought only in effort to advance the belief
system).
3. _____ Height of power, or period when this leader exercised the most influence on others, was during
designated time-frame (if not, please write in inclusive years and justification).
Fig. 1. Example scale for non-leadership SMEs to use in validating sample cell assignments.
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
1121
terrorist leaders' psychological profiles, Sageman (2008) gathered and coded open-source reports of identified leaders of violent
groups under the Global Salafi Mujahideen ideology. Sageman, a trained clinical psychologist who now works closely with the
intelligence community, was well-suited to identify and quantify evidence of varying personality characteristics of terrorist leaders
via secondary source descriptions. However, his ratings were singular in nature; no effort was made to assess the reliability of the
ratings he assigned. This clinical, case study approach makes it difficult to replicate findings across researchers, limits the applicability
of this approach given the burden on the PI's time, and provides limited reliability and validity evidence for inferences made. While
the coding by a single domain expert is useful in efforts for theory development because it yields rich contextually anchored
information, for historiometric analysis of leadership in general we recommend training and using multiple coder teams to gain an
index of inter-rater reliability for constructs observed. This type of training and set of procedures will be explained in greater detail in
section five, Coding Logistics.
Second, we advocate for coding schemes to be developed with the same practices used in test development (Osterlind, 1998). That is,
constructs of interest should be defined, iteratively reviewed, and edited by a subset of SMEs in item writing to ensure clarity, parsimony,
and unidimensionality. For example, in a study examining the communication strategies of outstanding leaders, Mumford, Bedell, et al.
(2006); Mumford, Gaddis, Licuanan, et al. (2006); Mumford, Gaddis, Strange, et al. (2006); Mumford, Scott, et al. (2006); Mumford,
Strange, et al. (2006) were careful that item stems were generated to measure affect and use of stories as separate constructs, and only
assessed in communications with the mass public (see Fig. 2 for how these scales were finalized). This was refined via SME feedback
during the item generation phase of this project; earlier iterations were worded, “To what extent does the leader use emotionally
evocative stories or analogies to illustrate a point?” This first item was essentially double-barreled, in that some leaders used emotion but
not stories, and vice versa, to communicate with the mass public.
It follows, then, that validated scales used for other purposes can be adapted to these studies as another unique approach. For
example, in an effort to assess the leader–member exchange (LMX) relationships of outstanding leaders and their top
management teams (TMTs), Strange (2004) used both internally developed items and an externally validated scale. First, using
SMEs familiar with item-writing and LMX theory, Strange and colleagues developed a set of theory-driven (Dienesch & Liden,
1986; Graen & Uhl-Bien, 1998) content items for judges to apply to passages describing exchanges between leaders and followers.
Coders also applied a pre-established behavior description inventory, the LMX-7 (Schriesheim, Castro, & Cogliser, 1999), to
appraise the quality of leader-follower relationships by assessing these same exchanges. Results showed that personalized and
socialized leaders display significant differences on her study-specific relational items, as well as the LMX-7 standard behavioral
markers. In addition, using this multi-tiered assessment technique, Strange (2004) also found that relationship strategies such as
support, team-based, and power-sharing were related to metrics of sustainable performance.
Third, behavioral exemplars, benchmarked against the sample at hand, are used to illustrate differing levels of each construct for at
least two reasons. First, the nature of biographical material used in historiometry is heterogeneous, ill-defined, and unstandardized
given that it was prepared ipsatively by multiple scholars in diverse domains. To infer levels of psychological constructs (e.g., positive
affect), observable and concrete examples must be provided to coder teams as a reference point to facilitate reliable, valid ratings.
Second, outstanding leaders operate in unique environments, and thus the way their behaviors manifest should be operationally
defined by exemplars from other outstanding leaders. This approach, the development of benchmark scales, requires an initial
oversampling from the population of interest (e.g., CEOs) to array behavioral examples in a way that differentiates low, moderate, and
high levels of a given construct. We recommend oversampling by 10%, and then randomly selecting 10% of that sample to a priori
identify benchmark behaviors to exemplify constructs. Regardless of whether a Likert rating scale or a Q-Sort strategy (McKeown &
Thomas, 1988) is used, these behavioral examples can provide a sample-bound reference for raters to use in assessing leadership
constructs relative to those of other outstanding leaders. These are simply illustrative examples of behaviors demonstrative of the
construct of interest and can increase reliability indices of agreement between raters significantly when implemented. We
recommend using these benchmarks in rater training, but also including them on rating materials to be used in independent coding
by rater/judge teams.
Finally, in line with best practices of scale development, we suggest that each construct be measured by multiple items to
provide some index of reliability (Traub, 1994). There are two approaches we have found successful to do this: 1) multiple
Likert-scaled items for each construct, and 2) a modified Q-Sort approach. For the first approach, and similar to test item
development, we recommend a coding scheme where raters read through sampled biographical material and then make ratings
on multiple (at minimum four) Likert scales assessing related but orthogonal indices of the construct of interest. For example, to
assess the long-term impact of communication strategies, raters assessed six components of speech criteria such as 1) extent
people still quoted leader speeches, and 2) degree to which speeches are considered landmark events (Mumford, Gaddis,
Licuanan, et al., 2006; Mumford, Gaddis, Strange, et al., 2006). Items can then be empirically combined to create scale-level
assessment (e.g., sustainability of speech impact), granted that appropriate inter-item correlations are obtained (Traub, 1994).
While the generation of multiple items across constructs is a well-established way to transform qualitative information into
quantitative ratings, we have also found success using a modified Q-Sort technique (McKeown & Thomas, 1988). When there are
copious data points to be evaluated independently, Q-Methodology results in a reliable and unique way to obtain metrics of
leadership characteristics. Q-Sort techniques are hardly innovative (Stephenson, 1935), but historically they have been applied to
assess an individual's subjective viewpoints or arrayed personality markers (e.g., “this trait is most like me versus that trait”).
Once viewpoints are modeled into Q-Sorts, data analysis then examines N Q-Sorts (i.e., the individual/Q-Sort arrays, not the items,
serve as the unit of analysis) through some empirical reduction technique. There are multiple ways (e.g., factor analysis, cluster
and then discriminant function analysis) these can be analyzed to obtain factor scores to represent “typical patterns of
responding,” and then each sample member can be assigned a score on such factors or functions.
1122
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
While traditional social science research applies Q-Methodology to study constellations of traits, beliefs, or some other subjective
characteristic of individuals participating in the Q-Sort, we have applied it in an effort to analyze historiometric data via trained judges
as well. In a study examining the developmental themes related to the CIP model of leadership and the personalized versus socialized
variants of each, we used models from the adult development literature (McAdams, 2006; Pillemer, 2001) to sample seminal
developmental events that occurred in 120 leaders' lives from age 14-early career (Ligon et al., 2008). Our approach yielded roughly
1400 events across the leaders under consideration, and Table 3 illustrates each of these types of events.
The next part of this study required thematic coding of markers related to charismatic, ideological, or pragmatic leadership styles
in each of these events. After reviewing the relevant leadership literature about each of these styles and likely developmental themes
related to each, 28 dimensions were retained to differentiate charismatic, ideological, and pragmatic leaders' early developmental
experiences (e.g., risk taking, spirituality, and flexible decision making dimensions, respectively). Each of these dimensions was
then illustrated by benchmark behavioral examples. A team of six judges was asked to review each event drawn from the biographies
(e.g., career originating event) and assign the event to the thematic dimension categories. Specifically, judges were asked to
determine whether each event did or did not reflect relevant thematic content for each of the 28 dimensions under consideration.
Scoring then occurred by determining those dimensions to which the majority of the judges assigned an event. Dimension scores
were obtained for each leader by determining the number of events identified for a leader assigned to a given dimension, and then
dividing by the total number of events identified for the leader under consideration.
An example of an anchoring, or belief formative, event from Fidel Castro that scored high on ideological and personalized
leadership dimensions can be seen on Table 3. This event scored highest on thematic dimensions of Belief Commitment, Inspirational
Communication, Risk-Taking, Power Motives, and Negative Life Themes, as the majority of judges assigned this event to these
categories over others. This type of formative life event might help explain how Fidel Castro has been characterized as a personalized
ideological leader in multiple studies of his leadership exchange relationships (Strange, 2004), performance (Mumford, Strange, et al.,
2006), and problem solving style (Mumford, Bedell, et al., 2006). More detail about best practices in training these judges and the
overall rating procedure will be described in the following sections on Coding Logistics and Preparation of Materials. To the best of our
knowledge, the study of developmental events is the only historiometric one that has employed a Q-Methodology, but it seems
appropriate for any study of leadership where the assignment of multiple, relatively standardized activities or events to a large
amount of thematic dimensions is required. For example, McCall's (2010) work on experiential development activities could make
use of sorting experiences such as job rotations, project assignments, and mentoring into thematic dimensions of leadership skills
developed.
Using these best practices of treating leadership historiometric coding schemes with the same consideration required of
psychometrically sound tests and scales in general, there are three general categories of content to be coded across all historiometric
studies: predictor, criterion, and control variables. Model specification dictates what types of predictor content should be coded, and
Table 2 illustrates that constructs of interest range from Machiavellianism markers (O'Connor et al., 1995) to indices of morality
(Thorndike, 1936). In short, there is no recommended set of content to be coded in predictor data of historiometric leadership studies;
instead, if one follows the above described procedures and generates sample benchmarks of behavioral exemplars for each construct
of interest, virtually any leadership theory can be assessed in this type of research paradigm.
Related, there is much variability in the types of criteria assessed in these studies, depending on the theoretical model of
interest. For example, study-specific criteria assessed in Simonton (2001, 2006), while McCann (2005). In addition to these
study-specific criteria, it is recommended that a set of general performance criteria also be assessed to compare leaders across
domains and provide some evidence about the relationships between our theories of leadership (e.g., authentic leadership facets)
and performance in a social domain.
Based on findings from Strange and Mumford (2002), there are 12 general performance criteria that can be assessed regarding
overall societal impact. These have been used in numerous historiometric studies to differentiate leaders on short and long-term
performance metrics across occupations. The first five criterion measures are based on observations typically found in the
Expression of Positive Affect—To what extent does the leader express positive emotions (e.g.,
happiness, joy) in his or her speeches and communications with the mass public?
1 –The leader never expresses positive emotions in his/her speeches or communications with the mass
public.
3 – The leader occasionally expresses positive emotions in his/her speeches or communications with the
mass public.
5 – The leader frequently expresses positive emotions in his/her speeches or communications with the
mass public.
Use of Stories and Analogies—To what extent does the leader communicate his or her points through
the use of stories and analogies in his or her speeches and communications with the mass public?
1 – The leader never uses stories or analogies his/her speeches or communications with the mass public.
3 – The leader occasionally uses stories or analogies his/her speeches or communications with the mass
public.
5 – The leader frequently uses stories or analogies his/her speeches or communications with the mass
public.
Fig. 2. Example items and rating scales for positive affect versus the use of stories.
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
1123
prologue chapters of academic biographies, and are quantified via counts examining 1) the number of positive contributions, 2)
the number of negative contributions, 3) the number of different types of positive contributions, 4) the number of different types
of negative contributions, and 5) the number of institutions established by the leader. What is helpful about these criteria in a
content coding scheme is that they require raters to make judgments about relatively objective markers on metrics that should be
applicable across the types of eminent individuals in historiometric leadership samples, and information can be readily verified
across multiple reference sources. In addition to these counts of criteria mentioned, a SME in leadership applications or a trained
judge (if provided with behavioral benchmarks) is required to make ratings on seven additional criteria after reviewing
biographic material. Table 4 illustrates the items that these ratings examine, made on a 5-point Likert scale.
The final type of items to code in leadership historiometry is intended to provide covariate controls for the inferences being
drawn. Similar to the predictor and criterion content coding schemes, each exemplar study in our review applied study-specific
controls. For example, when examining LMX relationships with leaders in a dissertation about leaders and top management
teams, a number of follower-demographic control measures (e.g., similarity of followers to leader) were formulated to
examine the inferences' generalizability to other LMX relationships (Strange, 2004). In addition to these particular content
analysis considerations, we identified a number of additional measures meant to assess internal and external validity concerns.
These constructs were assessed either via objective counts (e.g., word count), ratings on Likert scales (e.g., amount of
documentation provided for objective markers), or categorical assignment (e.g., was the leader a pre- or post WWII leader?).
To control for threats to source quality or internal validity, material characteristics regarding author credentials/training,
amount of data available (and amount of secondary supporting sources used to obtain that data), and degree of media and
author bias are some of the most commonly applied items. To assess external validity concerns, a series of measures are applied
to assess the leader's role and the context in which he/she led. For example, most studies controlled for type of leadership role
(e.g., business, political, military, non-profit), number of years in leadership power, and time-frame and region of leader's
height of power. More detail will be provided in the section on analyses as to varying ways to examine and account for these
variables' influence on the relationships of interest. For a full list of recommended internal and external validity controls, see
Table 5.
Table 3
Examples of developmental life events used in Q-Sort.
Event type
Benchmark example
Originating event
“From an early age, the young Rupert [Murdoch] was aware o the power and the glory and the sheer fun which accrued
to his father from newspapers. Keith [Rupert's father] used to take his son around the Herald's office on Flinders Street,
and Rupert often said later that the smell of the ink, the noise of the presses and the highly charged atmosphere were
irresistible. ‘The life of a publisher is the best life in the whole world. When kids are subjected to it there's not much
doubt they'll be attracted to it.’” (Shawcross, 1997 p.27)
“The most dramatic story concerns Lewis's involvement in the 1903 disaster at the Union Pacific Railroad Company's coal
mine in Hanna, WY. Passing through the area by chance, Lewis arrived in time to assist a rescue team in carrying out he
torn, charred bodies of 234 miners…‘what ripped his emotions to shreds was the sight of the numb, mute faces of the
wives now suddenly widows of the men they loved.’” (Dubofsky & Van Tine, 1986 pp. 14–15)
“In what Fidel calls, ‘a decisive moment on my life,’ Angel Castro decided during the boys' summer holiday after the 4th
grade that they would not go back to school…But Fidel [Castro] was determined to return to school. As he tells the story,
‘I remember going to mother and explaining to her that I wanted to go on studying; it wasn't fair not to let me go to
school. I appealed to her and told her I would set fire to the house if I wasn't sent back…so they decided to send me back.
I'm not sure if they were afraid or just sorry for me, but my mother pleaded my case.’ Fidel was learning quickly that
absolute and uncompromising stubbornness was a powerful weapon. This may have been the most important lesson he
had drawn from his young years at the finca, and he never forgot it.” (Szulc, 1986 p. 112)
“Almost forty years later, on the occasion of a commencement address at Fisk, and perhaps under the influence of the
occasion, DuBois recalled those three years of “splendid inspiration” and nearly “perfect happiness” with teachers whom
he respected, amid surroundings which inspired him. The ten years after Fisk he chronicled as “a sort of prolongation of
my Fisk college days.” I was at Harvard, but not of it. I was a student of Berlin but still a son of Fisk. I used my days there to
understand my new setting…” (Broderick, 1959 p. 9)
“She [Betty Friedan], who had been the ringleader and chief instigator, the one who generated all the excitement, was
suddenly alone, abandoned by her friends. The creator of clubs was not chosen for the most exclusive club at all—the high
school sorority. She was desolate…The year of loneliness that followed was the lowest point of her life. She blamed it
primarily on anti-Semitism…The sight of the car full of friend, a vision that she yearned for, triggered something in her,
and she made a promise to herself: ‘They may not like me now, but [someday] they are going to look up to me.’”
(Hennessee, 1999 p. 15)
[After receiving average marks on his officer's appraisal, [Charles] de Gaulle was given a lackluster assignment.] “Indeed,
for a soldier with his innate conviction of his intellectual superiority, the choice of a department concerned with such
routine matters as transport and supply was humiliating. At Mayence, in fact, he was put in charge of refrigeration, which
must have seemed an insulting punishment for an unwelcome independence of spirit…[de Gaulle after receiving the
news] ‘Those c…s of the Ecole de Guerre! I shall only come back to this dirty hole [sale boite] as Commandant of the
Ecole! And you'll see how everything will change!’” (Crozier, 1973 p. 39)
Turning point event
Anchoring event
Analogous event
Redemption event
Contaminating event
Adapted from Ligon et al., 2008.
1124
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
4.4. Materials
As historic records provide a rich, contextually-anchored source of descriptive data, successful application of the coding
schemes described in the preceding section depends to a large degree on the procedures used to draw behaviors and apply ratings
accordingly. In addition, depending on the type of research questions addressed, care must be taken to abstract relevant samples
of behavior. In a study examining how political leaders used masculine rhetoric and ideologies as a political strategy, Coe, Domke,
Bagley, Cunningham, and Van Leuven (2007) selected direct quotes from George W. Bush from a sample of communications
found in the National Archives and Records Administration's Weekly Compilation of Presidential Documents. In Simonton's
(1998b) examination of King George III of Great Britain's possible precipitators of poor health, he reviewed academic biographies
of King George III and sampled only references to the king's physical and mental health to be coded for stress and health ratings.
Finally, to assess Alan Greenspan's changes in communication style in relation to differences in the economic environment, Bligh
and Hess (2007) applied content analysis to testimonies and speeches available on the Federal Reserve Board's website.
Material to be content coded should be prepared in such a way so as to address three concerns with historical data:
1) reduction of bias from raters, 2) standardization of source material, and 3) ease of use of coding schemes for raters. As noted
earlier, the approach to leadership historiometry we recommend is one rooted in obtaining multiple ratings from multiple,
trained coders. Thus, a first concern when conducting this type of research is to prepare materials in a way that minimizes bias
from raters. As a result, when preparing abstracted source material from historically notable leaders, by definition some caution
must be applied to limit coder bias. In our studies of the 120 eminent leaders (e.g., Malcom X, Rupert Murdoch, Michael Collins),
rater teams were assessed for bias by asking for confidence ratings from coders as to their a) degree of familiarity with each leader
and b) knowledge of positive versus negative reputation of each leader. After numerous trials with our raters, the overwhelming
majority of whom were undergraduate psychology majors, we found that fewer than 10% of the leaders were consistently and
accurately recognized. While this is somewhat of a disheartening finding in general, for the purposes of reducing rater bias we can
be somewhat certain that leaders in our sample were unlikely to be rated with systematic bias given the lack of familiarity with
eminent historical leaders in general from undergraduate college students.
However, we recommend taking similar precautions to assess rater bias prior to beginning any historiometric study, and
particularly so when exemplars at hand carry strong cultural recognition (e.g., Osama bin Laden). In the event such bias is detected,
preparing data in a way that removes identifiers is recommended. For example, we conducted our study on outstanding leaders at the
same time that Martha Stewart, one of our sample members, received increased media attention due to felony convictions and a
string of television documentaries depicting Stewart in an unfavorable light (Ligon et al., 2008). To reduce potential bias, individually
identifying information about her and all leaders in our sample was subsequently removed from material to be coded.
Second, as biographic information used in historiometric studies is complex, situationally contextualized, and prepared by
different authors, standardizing how behaviors are sampled and presented to rater judges can increase the reliability of coding. One
way to do this is to create profiles for each type of event or behavioral example for a given leader. Simonton (1986a, 1986b, 1988a,
1988b) has created leader profiles with various types of biographical data that have been used in numerous studies (e.g., Deluga,
1997, 1998), and developmental event cards were generated for the Q-Sort procedure for the study on developmental patterns (Ligon
et al., 2008). This extra step of collating and re-organizing content from fact books, biographies, and other sources is time consuming
and requires careful attention to detail to ensure judges are able to evaluate all relevant material. However, by providing materials to
be rated in a standardized fashion, raters are able to reduce coding time for the study and easily find information upon which to assign
ratings. Thus, this approach is recommended when a study employs relatively unskilled raters rather than domain experts who might
be more readily able to assess relevant variables in copious, unstructured data.
Table 4
Typical performance criteria used in historiometric leadership studies.
Count metrics
1. Number of positive contributions made by the leader.
2. Number of negative contributions made by the leader.
3. Number of different positive contributions made by the leader.
4. Number of different negative contributions made by the leader.
5. Number of institutions established by the leader.
Likert scale rating metrics
6. How much did the leader contribute to society?
7. How long did these contributions last?
8. How many people did the leader affect?
9. Did the leader initiate mass movements?
10. Was the leader's agenda maintained when he/she left power?
11. Were institutions established by the leader still in existence?
12. What was the biographer's evaluation of the leader?
Note. A great deal of empirical work has been executed to demonstrate the validity and reliability of the
procedures used to apply these metrics to studies of outstanding leadership. For more detail about this
validation effort, consult Pathways to Outstanding Leadership, by
M.D. Mumford (2006, pp. 71–73).
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
1125
Finally, most of the coding schemes illustrated in Table 2 required judgmental evaluations on multiple different types of
constructs. Assigning ratings is cognitively taxing, and assigning ratings on multiple different types of indices can be exhausting and
laborious for judges. Thus, we recommend a) keeping rating anchors scaled in a consistent fashion, as there is limited evidence that
varying types of response options (e.g., rotating high versus low anchors) provides any psychometric benefit (Osterlind, 1998),
b) providing benchmark examples on or near rating scales for ease of reference, c) limiting the number of assessments any one rater
makes per event/behavioral example in a given session (e.g., for the Q-Sort we did, pilots indicated that raters could reliably sort up to
50 events into thematic categories in a 2-hour rating block, more or less than that resulted in decreased reliability), and d) organizing
all materials fastidiously, either through electronic means or hard copy bindings. This last step, material organization, is often
overlooked in project plans of these studies, but our experience indicates that if done well, this step can take graduate student teams
up to 2–4 weeks, depending on the sample size and number of coding dimensions.
5. Coding logistics
Related to the above section, raters must be trained extensively on how to use these materials in a reliable fashion. In addition, a
general framework should be developed to implement the coding process, as multiple independent rater teams are recommended for
predictor, criterion, and control dimensions. Thus, in the following section, we will provide detail about such coder training programs,
specifically benchmark and calibration training, assessing reliability, and developing a system for rater teams to code material.
First, content coders should be comprised of teams separate from those who gathered the materials to be coded. In our work,
we have found success using a mix of graduate and undergraduate students pursuing degrees in psychology, but given the
training that occurs for these judges, individuals from any major could be successful in this vein. In considering numbers, it is
recommended that at least three judges make each rating that requires some level of content judgment (e.g., Q-Sorts, Likert
ratings), but for metrics that are more straightforward in nature (e.g., controls such as word count), one judge should suffice.
After recruiting judges, we recommend a two-week benchmark training program, which typically is comprised of 4–6 h of
face-to-face training, and 4–6 h of independent work by judges. In this program, judges are first familiarized with the nature of the
stimulus material (e.g., developmental events, problem solving descriptions, or leader–member exchanges) that will be used in coding.
Next, they are presented with operationally defined dimensions which they will code in this material. For example, in Eubanks and
colleagues' (2010) study on how leaders responded to criticism, judges were familiarized with definitions of collaborative versus
confrontational strategies, and then raters were asked to restate in their own words what key concepts differentiated these approaches.
As situating these definitions in the sample at hand is critical for leadership historiometry, the next step in training is to illustrate
varying levels of these constructs with benchmark examples abstracted from the population of interest (e.g., U.S. Presidents). This
serves two purposes. First, it allows judges to better understand how these expected behaviors manifest in this type of sample,
providing a framework to use in comparing similar behaviors from individuals in the study at hand. Second, the use of these
benchmarks in training helps judges learn which information is useful for coding (i.e., concrete behavioral descriptions) versus which
information should be discounted (e.g., superfluous contextual details, author subjective statements). In short, benchmark training
facilitates raters learning how to assess only the incident under consideration when making ratings.
Rater training should also cover typical biases (e.g., halo, leniency, central tendency errors) and specific errors made in
historiometric studies (e.g., focus on evaluative/subjective statements, imposition of personal biases, and inadequate attention to
nuances of academic writing). Following this informational portion of training, rater teams should practice assigning a small
subset of ratings in the session with others and the study PI. When ratings are made on a 5-point scale, we recommend discussing
differences greater than 2 points. This type of synchronous, real-time discussion of subtle distinctions using the same dataset can
increase the reliability of ratings substantially.
Table 5
Typical covariate control measures applied in historiometric leadership studies.
Internal validity controls
1. Word count.
2. Number of outside sources referenced.
3. Author education level.
4. Author credibility (e.g., multiple publications on leader, graduate training, awards won).
5. Degree of bias (both positive and negative biases should be assessed).
6. Was author raised in subject's (leader's) country of origin?
7. Primary type of publication (e.g., peer reviewed).
8. Amount of objective predictor data available.
9. Amount of objective criterion data available.
10. Was source translated?
External validity controls
11. Geographic region of leader (i.e., Western or Non-Western).
12. Development of leader country during height of power (i.e., industrialized or non-industrialized).
13. Munificence of leader country.
14. Post or pre-WWII time period.
15. Length of height of power.
16. Type of organization (i.e., for-profit, military, government, non-profit).
1126
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
Next, raters are assigned a small subset of leader behaviors (predictor, criterion, or control data) to independently assign
ratings over the next week. For Likert items, using procedures outlined by LeBreton and Senter (2008), the study PI then can
assess inter-rater reliability of these ratings. For categorical assignments (e.g., Q-Sort procedures), a kappa agreement coefficient
should be obtained to assess inter-rater agreement of categorical assignments (McKeown & Thomas, 1988). Given that raters are
asked to make assessments about a large amount of variables on relatively complex information, we have found that inter-rater
reliability coefficients typically range from .70–.90, and kappa coefficients range from .55–.90 for well-calibrated raters. Until
agreement indices fall into these ranges, however, we recommend repeating these iterations of training (particularly the discussion of
differences larger than 2 points) with content coders. In addition, at several points during the coding process (particularly for large
samples), we have found success with taking “pulse” reliability and agreement assessments. Specifically, we recommend assessing
reliability and agreement at 10% increments of the coded material (so for N = 100, assessing coders at n = 10, 20, 30 leaders). When
coefficients of agreement fall below acceptable ranges, we recommend conducting “calibration training,” or training intended to
re-orient raters to how each other see the differences in the data.
In making these Likert ratings, judges are typically presented with either a binder or electronic folder containing a subset of
the relevant stimulus materials abstracted from the historical records. The material should be structured in a way where material
applying to one leader is distributed across multiple binders or folders and thus not expressly identifiable for the coding teams. In
addition, the rating plan should ensure that these binders/folders are rotated across the judges so that each leader was assessed
by various coders at different levels of training throughout the process. For the Q-Sorts, cards to be sorted (e.g., non-identifiable
descriptions of developmental events, Ligon et al., 2008) were also distributed in a way that raters were not sorting all cards of
one leader in a given session.
Once all of the data has been coded and acceptable reliability coefficients are obtained, the three independent coders' ratings
should be averaged to lessen likelihood of spurious errors in any one individual's scores. For Q-Sorts, assignments can be either
averaged across judges or added together to obtain arrays for each unit of analysis. In the following section, we will describe how
these metrics are typically evaluated using descriptive and inferential analytic frameworks.
5.1. Analyses
Like other methodological best practices, the analyses flow directly from the theoretical model under examination. If the
above-detailed practices to gather multiple predictors, criteria, and controls are implemented, the research team is left with a
collection of quantitative data to which virtually any analytic technique can be applied depending on the research model
specified. Given that multiple analyses exist for historiometric data, the following section is not intended to be prescriptive in
nature. Instead, in line with the other sections, we will provide one exemplar set of techniques we have found useful, particularly
when handling the internal and external validity controls.
When the theoretical model specifies an examination of the patterns of individual differences that drive distinctions between
types of leaders (e.g., the CIP model) and performance, a multi-tiered analytic approach based on multivariate analysis of covariance,
discriminant-function analysis, and regression can be useful (Mumford, 2006). After running appropriate exploratory statistics to
examine (i.e., check for assumption violations) and clean the data (i.e., correct data entry errors), the first set of analyses are typically a
series of multivariate analyses of covariance (MANCOVAs). In these analyses, the grouping variables of interest (e.g., personalized
versus socialized leader orientations) are treated as independent variables and the dimensions under examination (e.g., degree of
collaboration, scores on Big Five personality variables, jargon used in speeches) are treated as dependent variables. The covariates
applied in these analyses should include both internal and external validity controls, and retained if those covariates proved
significant (p b .05) with the dependent variables of concern and assuming they met the sphericity assumption. In short, inferences
about differences (e.g., use of jargon) across grouping variables (e.g., leader orientation) are only made after taking into account
important controls.
The next set of analyses provide a summary description of how leader groups (e.g., personalized versus socialized, assassinated
versus not assassinated) differ from each other in terms of the dimensions under examination (e.g., LMX techniques used).
Thus, when even marginally significant differences (p b .10) are obtained across independent variables in the MANCOVAs, a
discriminant-function analysis can be conducted to identify the underlying set of variables that best account for these differences. This
approach allows researchers to assess packages of variables, as many of these operate in tandem to describe behaviors in domains as
complex as leadership (Klecka, 1980). For example, in a study examining problem solving differences of charismatic, ideological, and
pragmatic leadership styles, a significant main effect was obtained from the MANCOVA analysis when taking into account general and
study-specific controls [F(8, 102)= 33.63; p b .001]. As a result, a set of discriminant functions were obtained to distinguish
charismatic, ideological, and pragmatic leaders, and dimensions yielding sizable loadings on that function were information gathering
(r = .75), concept selection (r = .42), and idea generation (r = −.35). Because the dimensions underlying this function represented
information acquisition, this function was labeled Expertise by the research team (Mumford, Bedell, et al., 2006).
The discriminant functions obtained in this second set of analyses provide the basis for the third analytic step: obtaining leader
scores on each of these functions. In the above-described example of problem solving, average scores were obtained for each of the
three leader types. Pragmatic leaders, known for their analytical approach to leadership (Mumford & Van Doorn, 2001), scored the
highest on this Expertise function (M = 1.57) when compared to charismatic (M = −1.73) and ideological leaders (M = .15). Thus,
this third step in the analytic framework can illustrate how the different groups of leaders (e.g., styles, assassinated versus not) score
relative to important differentiating characteristics.
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
1127
The final set of analyses is designed to examine how these important differences relate to leader performance or some separate
criterion of interest (e.g., health ratings). To do this, each leader's scores on a given criterion variable must be obtained (e.g., number of
positive contributions). Each criterion score is then correlated with and regressed on the scores on the discriminant-function
variables intended to summarize cross-group differences in dimensions under examination from steps 1–3 (e.g., the scores on the
Expertise function in the previous example). When conducting these regressions, however, it is important to note that significant
(p b .05) control variables identified in the first set of MANCOVAs are entered as the first block of predictors. This helps to ensure that
observed differences take into account important internal and external validity concerns by covarying out their influence statistically.
While these four steps represent a general framework for how to reduce and interpret the data obtained in leadership
historiometry, other techniques associated with study-specific differences can also be incorporated into this scheme. For example,
Simonton's (2010) review of time series approaches to examine these variables is quite detailed. Another example can be found in
Aldenderfer and Blashfield's (1984) publication, which reviews techniques used in cluster analysis to investigate useful coding
schemes and taxonomies. This last approach holds particular promise when the research question is more exploratory in nature
than the ones summarized in Table 2.
6. Summary for methodological best practices
In the above six sections, we reviewed contemporary approaches for conducting historiometric studies of leadership. When
reviewing the studies presented in Table 2, there are three key lessons that can be taken from the scholars involved with this
research. First, sampling should be approached rigorously so that inferences can be made about not only outstanding leaders in
the population of interest, but also for leadership theory development in general. Studies that sample from heterogeneous
populations and control for cross-group differences (e.g., industrialization of country led, munificence of organization) are more
readily generalizable to individuals outside of positions of “outstanding” influence.
Second, content coding should be approached similarly to test item development, with a focus on the generation of items and
benchmarks that allow for assessments to be made on objective, behavioral data. Rather than making subjective judgments based
on historical data that is often variable in focus, length, and detail, we delineated a set of item generation and scale validation
procedures, based on what is available in the broader psychometric literature, that can be used to translate the qualitative
information about these leaders into metrics to be used in multivariate statistical techniques to draw valid, reliable inferences. In
addition, a plan to gather and measure indices to statistically control internal and external validity concerns is encouraged given
the variability found in historical source data in general.
Third, leadership historiometry requires multiple steps, types of personnel (e.g., PI, SMEs, coders), and resources (e.g., biographies,
external criteria). A detailed project plan should be defined early in the process. If PIs are aware of sequence and logical ordering of
these steps, however, these studies, like other paradigms in the social sciences, can take less than a year to execute from start to finish.
It is our hope that defining these methodological best practices in detail for this present effort will facilitate other leadership
researchers to use and share this technique in their own programs of research. However, before embarking on this type of study, a
careful review of the advantages and disadvantages associated with it should be evaluated. In the following section, we outline a set of
pros and cons for using historiometry to study leadership.
7. Evaluation of the technique for assessing leadership: advantages and disadvantages
Like any empirical technique, historiometry comes with pros and cons for the study of leadership. There are five noteworthy
concerns for this type of research paradigm: 1) generalizability of findings, 2) empirical rigor and reliance upon secondary
sources, 3) heterogeneity of performance measures, 4) intensity of labor involved, and 5) inability to draw causal inferences.
Before discussing the advantages, we will turn to considerations associated with each of these oft-cited criticisms.
7.1. Generalizability of findings
First, by definition historiometric techniques require examination of leaders whose influence is historically notable. As such,
they tend to work in pinnacle leadership roles in domains such as government (e.g., Barack Obama), corporate (e.g., Warren
Buffett), military (e.g., Dwight Eisenhower), and non-profit and/or religious settings (e.g., Indira Gandhi). This poses a problem on
at least two fronts. First, the problems which they face in leadership roles are likely more complex, ill-defined, and critical than
what might be expected in other leadership roles. Thus, the skills required to execute them might be unique to those instances
(resulting in low base rates of occurrence in general). While this is certainly a concern when generalizing from studies on more
normative leaders to eminent leader populations, it causes less concern as understanding phenomenon in higher-ordered settings
can help to infer what behaviors, albeit perhaps lower levels of such behaviors, might prove useful for leadership in general. For
example, in the study on outstanding collegiate and NFL football coaches, we found the expression of ideological leadership was
related to performance of teams at both levels. While leading a high school football team may not require the same degree of
leadership skill and exact same behaviors, it is a logical assumption that coaches at these more conventional levels of leadership
power would benefit from using some of the same techniques to motivate their teams and coaches (e.g., reflection on past great
teams as exemplars for performance).
Another external validity concern revolves around the personnel surrounding these leaders; because these leaders are
likely to have direct reports or TMTs who are also comprised of high level leaders, it is difficult to infer that their performance
1128
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
(e.g., sustainability of influence) can solely be attributed to the leader alone. This may not be a similar condition when extending
findings to other types of more normative leaders, where a different set of techniques might be critical for performance. For example,
it is well-established in a host of empirical studies that pragmatic leaders tend to execute their agendas via elites, or key idea
champions and experts in their TMT and networks (Mumford & Van Doorn, 2001). It is difficult to generalize those influential
behaviors to leaders who might not have access to such high level leaders around them. Thus, some caution is encouraged before
drawing conclusions about leadership regarding the nature of how these leaders operate with and through collaborators.
7.2. Empirical rigor and reliance on secondary source material
As the unit of analysis requires the use of secondary sources and coding schemes rather than direct observations, the empirical
rigor of these studies can be questioned. Given that we are relying upon the social construction of the events leading up to and
during such sample members' height of power, we run the risk of making inferences based on the social structures and practices
as recounted by the biographers and historical writers rather than more objective, factual accounts from history (Boje, 2001;
Fairclough, 2003; Mills, 2003). For example, in an effort to appeal to the American Dream, early biographers of Walt Disney —
commissioned by Disney himself — recounted a self-made entrepreneur who fostered a climate of congenial creativity and
autonomy (Boje, 2001). Academic historians later described a leader who preferred a hierarchical organization, instilled fear in
meetings, and aggrandized his own contributions (Mosely, 1985). In essence, there are multiple realities, and they are each
subject to the intentions of the author and to some degree the raters (Boje, 2001).
This and other examples illustrate that the inferences made are only as valid as the procedures used to sample and assess
them. Thus, this criticism is a valid one if special care is not taken with respect to statistical and inferential concerns when
sampling, designing coding, and conducting analyses. As noted above, if researchers comport these studies with psychometric
principles as a guide (e.g., multiple measures, methods, and people), then this approach can be as rigorous as other paradigms to
gather similar inferences. As Simonton (2010) noted, “the scientific rigor of historiometric research is often underappreciated. Not
only can these measures display quite respectable reliability coefficients, but in addition the application of advanced multivariate
and quasi-experimental designs can help make some inferences more secure” (p. 282).
7.3. Contextual influences on performance measures
A related criticism of using historiometry to examine leadership lies in the social construction of performance measures.
Specifically, what is judged as effective leadership is bound by situation, time, and other contextual factors (Mills, 2003). Moreover,
the dominant view at the time of the analysis might impact the events, characterizations of the individuals' roles in them, and even
the net valence of a given leader's impact. In addition, because these leaders tend to manage large, complex organizations, it is difficult
to draw direct linkages to specific behaviors and performance outcomes. We have advocated applying controls to measure the
influence of contextual variables such as time, region, munificence, and gender of the leader, but more work needs to be done to
ascertain a covariate list specific to application of historiometry to leaders, and particularly in reference to the research question at
hand.
While we have argued for the need to obtain hard, objective data when possible through historiometry, we cannot lose sight of
what subjectivity and richness affords, especially when sampled carefully. As such, we advocate combining more subjective
accounts of performance (e.g., author descriptions of performance) with more objective indicators (e.g., number of institutions
established), as the social construction of what was outstanding performance can be a useful indicator referent to other leaders in
different time periods (Boje, 2001).
One example of the benefits of utilizing subjective and objective criteria can be found in a study conducted on the performance
contributions of eminent leaders of scientific endeavors (Mumford et al., 2005). Here we examined the academic obituaries of 600
eminent leaders in social, biological, health, and natural scientific domains. When examining the relationship between subjective
biographical accounts of performance and more objective indices (e.g., publication counts, impact factors, patent applications), we
found author “bias” (i.e., amount of positive hyperbolic descriptors of a given scientist's work contributions by the obituary author)
was related to objective performance indices in a given field than it was to other contextual factors (e.g., time period, obituary author
credentials). Most interesting is that whereas field norms differed in terms of objective performance indices (e.g., number of
publications differed for eminent scientists across natural versus social sciences), the obituary authors' subjective accounts of leader
contributions tended to be more stable across fields. Thus, it seems that subjective accounts, instantiated in what the dominant view
of “performance” was at a given place and time, may provide the most valuable source of performance information to compare across
domains (e.g., government, for-profit business, military) and time.
7.4. Intensity of labor required
There are three time-intensive steps associated with the procedures. First, sample selection and gathering requires the use of
external judges (e.g., SMEs), an internal logistics team (e.g., graduate students), and multiple trips to the library. For example, in
Mumford's (2006) sample of 120 leaders, originally 180 leaders were identified to fit the parameters of interest. Preliminary
research was then gathered about each of these individuals from multiple team members to assign them CIP categories. After
SMEs (an external team of productive leadership scholars) assessed each of these exemplars, the sample was reduced to 120
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
1129
leaders. However, in an effort to gain reliable information, two and sometimes three academic biographies were gathered for each
of these individuals.
This part of the process — the first step in executing a series of studies examining outstanding leadership — lasted three months
and required 10 h per week of three doctoral students. Second, rater training is an often overlooked step that dramatically increases
the reliability of judge coding. This step not only requires personnel to develop the training, but also a minimum of two weeks for
judges to go through the training. Finally, coding of the data, while it does have benefits for raters (e.g., learning about leaders and
applying well-established metrics to observe them), can be laborious and time consuming for rater teams. Because we recommend
rotating judges across different sets of variables, this step can be tricky to execute if not planned out thoughtfully. For the one year
duration of the studies in Mumford's (2006) book on outstanding leaders, twenty-four undergraduates and eleven graduate students
were used as coding teams for the various study predictor, control, and criterion data. While other empirical paradigms, particularly
studying intact teams in applied settings, can require similar types of investment, it cannot be underemphasized that leadership
historiometry is indeed labor intensive if done correctly.
7.5. Inability to draw causal inferences
As noted in Section 6 on analyses, the data that can be drawn from this type of technique is correlational in nature. As a result,
relationships that can be observed may indeed be caused by spurious, third-level variables. When conducting these studies, it is
critical to measure and statistically control confounding variables to provide some safeguards around this. However, similar to
other methodological approaches in the field of leadership (e.g., Hunter et al., 2007), it remains that leadership historiometry is a
technique that allows inferences about patterns of behaviors, events, and characteristics as they related to varying leadership
styles and performance.
The five issues identified are important to consider, but at least four advantages make historiometric research a highly viable
paradigm for our field: 1) theory testing, 2) real world generalizability, 3) availability of leadership observations, and 4) replicability
of procedures.
7.6. Theory testing
The most important contribution of historiometric techniques is the capacity to test theory. Given a rigorous, theory-driven
sampling procedure, historiometric techniques afford comparison between leader types, nations, organizations, time periods, and
domains in ways that augment other approaches in our field. For example, in an effort to assess variables accounting for the most
variance in presidential rankings, Simonton (2001) sampled U.S. Presidents and applied a coding scheme to assess degree of
scandal while in office, presence of war, heroism in war, and intellectual brilliance. Because he examined this across time periods,
he was able to adequately test the robustness of his theories of greatness across all members of the sample of interest: U.S.
Presidents. It is our argument, then, that historiometric techniques provide the best way to examine the robustness of our
theories of leadership across a host of settings.
7.7. Real world generalizability
Because historiometry is not conducted on convenience samples or in a laboratory, it allows us to examine leader behavior in
complex environments over variable conditions. And while we sacrifice some of the internal validity that assignment to carefully
designed conditions allows, the examination of leaders in actual organizations over time allows for inferences to be made to other
leader situations more readily. In addition, this type of methodology allows us to have case exemplars for illustrating leadership
principles in training and academic settings. As case-based training has been linked to enhanced learning from novices,
historiometric techniques might be a particularly fruitful approach for generalizing leadership principles and illustrating points to
students with actual behaviors. For example, the database on developmental events allows fast access of events depicting
important characteristics (e.g., presence of spirituality in seminal life events of ideological leaders like the one in Table 3 on Fidel
Castro) to use in illustrating similarities and differences among leadership theories (Ligon et al., 2008).
7.8. Availability of leadership observations
Life history data of eminent leaders are replete with copious objective markers regarding leadership predictors, controls, and
criteria. These markers range from descriptions of educational events that transpired, emotional expression in personal letters
and artifacts, to use of the word “I” as an indicator of self-aggrandizement in describing accomplishments (Deluga, 1997). In short,
this type of research allows scholars to study variables (e.g., aggression, war heroism) that may not be accessible in other
paradigms, given the sensitivity and rarity of their occurrence. In addition, because they are recorded in actual leadership settings,
their expression might have more range than one might expect in a laboratory or post hoc self-report study.
7.9. Replicability
Because a best practice in preparation of materials is to record and transcribe leadership data with precision, the type of
database afforded can yield exact replication of procedures and extension of findings. For example, after conducting our study on
1130
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
developmental influences on leadership performance (Ligon et al., 2008), Bedell and colleagues were interested to see how
important developmental influences predicted a range of other leader behaviors, such as problem solving, political tactics, and
leader–member exchange (Mumford, Bedell, et al., 2006). Given the accessibility of our database and events, she was able to
conduct a study with these new dependent variables measures with ease, following the procedures we had outlined. In addition,
Simonton's leader profiles have been used in several studies examining U.S. Presidents (McCann, 1992; Simonton, 1981, 1986a,b).
Thus, given the historical nature of sample material, this technique allows for enhanced and exact replication of results.
8. Conclusions and future directions
Before exploring the conclusions and future directions of leadership historiometry, it is important to note that there are at
least three limitations to this present effort. First, we chose to focus on a particular application of quantitative analysis of
qualitative data — that of outstanding individuals. Another related instantiation of this technique, for example archival data
analysis (Asal & Rethemeyer, 2008; Hepworth & West, 1988), may also be a useful tool in our repertoire to advance leadership
theory. While the majority of the procedures outlined would also fit archival analysis, one is cautioned about the quality of
information archival techniques afford. While historiometric research, and specifically the sampling approach we recommended,
focus on coding data from experts in their particular fields, archival data analysis often employ open-source documents such as
news reports, website posts, and even material from the organization itself (Sageman, 2008). When reading the reference section
of an academic biography, it becomes difficult to equate such source data to that found in some archival studies. For example, in
McCullough (2001). In addition, McCullough has received numerous investigative writing awards, such as the Pulitzer Prize for
his 1993 book, Truman. The use of such rigorous source information in sampling in historiometric studies may be one reason why
many of the internal validity controls prove non-significant (Mumford, 2006). Thus, if employing archival analysis on perhaps
non-outstanding sample leaders (e.g., emails, speeches from company presidents), a copious set of controls should be applied in
order to be sure findings are not due to some spurious third variable present in this type of non-verified data.
Second, we chose to propose a set of techniques, and specifically coding techniques rooted in psychometric theory, that the
present authors have found to be successful in this type of examination (Hunter et al., 2011; Ligon et al., 2008; Mumford, 2006).
Other techniques also provide useful, albeit different, types of information in biographic analysis. The application of validated
measures to assess personality characteristics (Simonton, 2006) or number of objective events (Simonton, 1997) provides rich
information about outstanding leaders. For example, in a time series analysis of eminent leaders in Japan, Simonton (1997) coded
the number of times such leaders traveled outside the country to examine the relationship between cognitive complexity and
exposure to outside influence. In addition, Simonton (2001). Whereas the approach we recommended, relying upon coding
multiple data-points and then reducing that data via multivariate analyses such as discriminant-function analysis, allows for the
examination of patterns of interacting markers related to performance, Simonton's approach facilitates obtaining important,
discrete markers that can be assessed independently to predict specific outcomes. His rigorous, thoughtful techniques have been
outlined extensively and should also be considered when deciding which approach to implement (Simonton, 2010).
Finally, while the intent of the present effort was to clearly specify the procedures needed for leadership historiometric
research, there are some areas we have not been as explicit in describing: Q-Sort, discriminant function, or time series analysis.
However, we have made an effort to provide the key resources we have found useful in conducting this part of our studies. For
example, the Sage Series in Quantitative Applications in Social Sciences has published editions on Q-Methodology (McKeown &
Thomas, 1988) and Discriminant Analysis (Klecka, 1980) that have proven invaluable in our work in this area. Each of these
resources provides specific details of the assumptions needed, analytic interpretations, and data requirements to execute them. In
addition, Simonton (1990, 2010) has published chapter-length pieces on best practices related to his approach, and specifically
for collecting time series data in historiometry.
Despite these and other limitations, the present effort has implications for leadership research. First, while there are numerous
leadership theories in general, few of them have been empirically tested on outstanding leaders. We are not arguing that this
technique should usurp traditional lab approaches or survey research, but instead hope that historiometry will be another
approach to examining the complex issues underlying high level leadership. We also hope this submission can aid researchers
interested in pursuing such endeavors. Related, another conclusion from this effort is that historiometric research is a multi-step,
labor-intensive approach, but also one that is relatively straightforward once executed. If sampling and content coding plans are
designed to obtain a comprehensive examination of the formative and height of power years of eminent individuals, resultant
data can provide the basis for a significant part of one's research program in leadership for multiple studies.
Given the richness of historiometric data and the rigor associated with data collection, there are at least four future directions
we hope to see employed. First, historiometric designs should be incorporated in validating leadership theory. For example,
models such as authentic leadership and complexity theory could be applied to eminent samples to better understand how these
leadership styles relate to performance, as well as how they differ from existing theory.
Second, recent advancements in text analysis should be investigated in conjunction with this approach. For example, Abe (2011)
used LIWC, a text analysis software program, to count changes in specific phrases or words in speeches of Alan Greenspan during
different economic periods. In addition, Bligh and Hess (2007) used a content analysis program called Diction 5.0 to assess the degree
of certainty, optimism, immediacy, and jargon on communications from Alan Greenspan during the same period. Using this
technology to assess frequencies of other types of psychological markers in larger, historiometric samples should be a way to assess
the robustness of these and other findings. We also recommend cross-validating existing content coding schemes (e.g., traditional
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
1131
coder teams) with such automated technologies to determine what kinds of research questions automated software can and cannot
reliably answer.
Finally, it is still unclear which internal and external validity concerns impact our capacity to draw inferences. Thus, our
recommendation is to collect those that we have delineated in this manuscript. However, this takes a considerable amount of time
to gather (e.g., ascertaining information about author education/training for each component of source material), and sometimes
even doubles the project timeline depending on how many sources per sample member are required for predictor and criterion
information. Thus, if historiometry is to become a tool used across a more diverse set of research questions and samples, it would
be beneficial to investigate which specific controls are most critical to covary out and for what types of criteria. As our hope is that
this type of research becomes more readily accessible to our field through this effort, further investigation of methodological
advances such as automated text coding and scaling covariate controls should help make historiometry a more commonly used,
widely understood approach to studying leadership in general.
Acknowledgements
Thanks to Mike Mumford who taught us just about everything we know on this method, even if doing so required far too many
weekend trips to the University of Oklahoma Library for all of us.
References
Abe, J. A. (2011). Changes in Alan Greenspan's language use across the economic cycle: A text analysis of his testimonies and speeches. Journal of Language and
Social Psychology, 30, 212–223, http://dx.doi.org/10.1177/0261927X10397152.
Aldenderfer, M. S., & Blashfield, R. K. (1984). Cluster Analysis (Sage University Paper Series on Quantitative Applications in the Social Sciences, series no. 007–044).
Beverly Hills, CA: Sage.
Asal, V., & Rethemeyer, R. K. (2008). The nature of the beast: Organizational structures and the lethality of terrorist attacks. The Journal of Politics, 70, 437–449.
Asal, V., & Rethmeyer, R. K. (2011). Wicked web: Social network analysis for combating Islamic and Arab terrorist networks. START Research Review, 12–13.
Avolio, B. J., & Bass, B. M. (1999). Re-examining the components of transformational and transactional leadership using the Multifactor Leadership Questionnaire.
Journal of Occupational and Organizational Psychology, 72, 441–462.
Baldoni, J. (2005). How great leaders get great results. New York: McGraw-Hill.
Bass, B. M. (1985). Leadership and performance beyond expectations. New York: Free Press.
Bass, B. M., Avolio, B. J., & Goodheim, L. (1987). Biography and the assessment of transformational leadership at the world-class level. Journal of Management, 13, 7–19.
Bass, B. M., & Farrow, D. L. (1977). Quantitative analyses of biographies of political figures. Journal of Psychology, 97, 281–296.
Bedell, K., Hunter, S., Angie, A., & Vert, A. (2006). A historiometric examination of Machiavellianism and a new taxonomy of leadership. Journal of Leadership &
Organizational Studies, 12, 50–72, http://dx.doi.org/10.1177/107179190601200404.
Bedell-Avers, K., Hunter, S. T., Angie, A. D., Eubanks, D. L., & Mumford, M. D. (2009). Charismatic, ideological, and pragmatic leaders: An examination of leader–
leader interactions. The Leadership Quarterly, 20, 299–315, http://dx.doi.org/10.1016/j.leaqua.2009.03.014.
Bligh, M. C., & Hess, G. D. (2007). The power of leading subtly: Alan Greenspan, rhetorical leadership, and monetary policy. The Leadership Quarterly, 18, 87–104,
http://dx.doi.org/10.1016/j.leaqua.2007.01.001.
Boje, D. M. (2001). Narrative methods for organizational and communication research. London: Sage.
Broderick, F. L. (1959). W. E. B. Dubois: Negro leader in a time of crisis. Stanford, CA: Stanford University Press.
Cell, C. P. (1974). Charismatic heads of state: The social context. Cross-Cultural Research, 9, 255–305.
Chadha, Y. (1999). Gandhi: A life. John Wiley & Sons, Inc.: United States of America.
Coe, K., Domke, D., Bagley, M. M., Cunningham, S., & Van Leuven, N. (2007). Masculinity as political strategy: George W. Bush, the "War on Terrorism," and an
echoing press. Journal of Women, Politics & Policy, 29, 31–55, http://dx.doi.org/10.1300/J501v29n01-03.
Cox, C. (1926). The early mental traits of three hundred geniuses. Stanford, CA: Stanford University Press.
Crozier, B. (1973). De Gaulle. New York: Charles Scribner's Sons.
Day, D. V. (2000). Leadership development: A review in context. The Leadership Quarterly, 11, 581–613.
DeChurch, L. A., Burke, C., Shuffler, M. L., Lyons, R., Doty, D., & Salas, E. (2011). A historiometric analysis of leadership in mission critical multiteam environments.
The Leadership Quarterly, 22, 152–169.
DeGregorio, W. A. (1991). The complete book of U.S presidents (2nd ed.). New York: Dembner Books.
Deluga, R. J. (1997). Relationship among American presidential charismatic leadership, narcissism, and rated performance. The Leadership Quarterly, 8, 49–66.
Deluga, R. J. (1998). American presidential proactivity, charismatic leadership, and rated performance. The Leadership Quarterly, 9, 265–292.
Deluga, R. J. (2001). American presidential Machiavellianism: Implications for charismatic leadership and rated performance. The Leadership Quarterly, 12,
339–363.
Dienesch, R. M., & Liden, R. C. (1986). Leader–member exchange model of leadership: A critique and further development. Academy of Management Review, 11,
618–634, http://dx.doi.org/10.5465/AMR.1986.4306242.
Dubofsky, M., & Van Tine, W. (1986). John L. Lewis: A biography. Urbana, IL: University of Illinois Press.
Eisenhardt, K. M. (1989). Building theories from case study research. Academy of Management Review, 14, 247–250.
Emrich, C. G., Brower, H. H., Feldman, J. M., & Garland, H. (2001). Images in words: Presidential rhetoric, charisma, and greatness. Administrative Science Quarterly,
46, 527–557.
Etheredge, L. S. (1978). Personality effects on American foreign policy, 1898–1968: A test of interpersonal generalization theory. American Political Science Review,
78, 434–451.
Eubanks, D. L., Antes, A. L., Friedrich, T. L., Caughron, J. J., Blackwell, L. V., Bedell-Avers, K. E., et al. (2010). Criticism and outstanding leadership: An evaluation of
leader reactions and critical outcomes. The Leadership Quarterly, 21, 365–388, http://dx.doi.org/10.1016/j.leaqua.2010.03.003.
Fairclough, N. (2003). Analyzing discourse: Textual analysis for social research. London: Routledge.
Fiol, C. M., Harris, D., & House, R. (1999). Charismatic leadership: Strategies for effecting social change. The Leadership Quarterly, 10, 449–482.
Gardner, H. (1993). Creating minds: An anatomy of creativity seen through the lives of Freud, Einstein, Picasso, Stravinsky, Eliot, Graham, and Gandhi. New York, NY US:
Basic Books.
Giambatista, R. C. (2004). Jumping through hoops: A longitudinal study of leader life cycles in the NBA. The Leadership Quarterly, 15, 607–624.
Gibbert, M., Ruigrok, W., & Wicki, B. (2008). What passes as a rigorous case study? Strategic Management Journal, 29, 1465–1474, http://dx.doi.org/10.1002/smj.722.
Graen, G. B., & Uhl-Bien, M. (1998). Relationship-based approach to leadership: Development of leader–member exchange (LMX) theory of leadership over 25
years: Applying a multi-level multi-domain perspective. In F. Dansereau, & F. J. Yammarino (Eds.), Leadership: The multi-level approaches (pp. 103–134).
Stamford, Connecticut: JAI Press.
Hennessee, J. (1999). Betty Friedan: Her life. New York: Random House.
1132
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
Hepworth, J. T., & West, S. G. (1988). Lynchings and the economy: A time-series reanalysis of Hovland and Sears 1940. Journal of Personality and Social Psychology, 51,
239–247.
Holmes, J., & Elder, R. E. (1989). Our best and worst presidents: Some possible reasons for perceived performance. Presidential Studies Quarterly, 19, 529–557.
House, R. J., & Howell, J. M. (1992). Personality and charismatic leadership. The Leadership Quarterly, 3, 81–108, http://dx.doi.org/10.1016/1048-9843(92)90028-E.
House, R. J., Spangler, W. D., & Woycke, J. (1991). Personality and charisma in the U.S. presidency: A psychological theory of leader effectiveness. Administrative
Science Quarterly, 36, 364–396, http://dx.doi.org/10.2307/2393201.
Hunter, S. T., Bedell-Avers, K. E., & Mumford, M. D. (2007). The typical leadership study: Assumptions, implications, and potential remedies. The Leadership
Quarterly, 18, 435–446.
Hunter, S. T., Bedell-Avers, K. E., & Mumford, M. D. (2009). Impact of situational framing and complexity on charismatic, ideological and pragmatic leaders:
Investigation using a computer simulation. The Leadership Quarterly, 20, 383–404.
Hunter, S. T., Cushenbery, L., Thoroughgood, C., & Ligon, G. S. (2011). First and ten leadership: A historiometric investigation of the CIP leadership model. The
Leadership Quarterly, 22, 70–91.
Jauch, L. R., Osborn, R. N., & Martin, T. N. (1980). Structured content analysis of cases: A complementary method for organizational research. Academy of
Management Review, 5, 517–527, http://dx.doi.org/10.5465/AMR.1980.4288954.
Kaarbo, J., & Beasley, R. K. (1999). A practical guide to the comparative case study method in political psychology. Political Psychology, 20, 369–391,
http://dx.doi.org/10.1111/0162-895X.00149.
Kempster, S. J. (2006). Leadership learning through lived experience: A process of apprenticeship? Journal of Management & Organization, 12, 4–22.
Kempster, S. J. (2009). How managers have learnt to lead: Exploring the development of leadership practice. UK: Palgrave Macmillan.
Kenney, P. J., & Rice, T. (1988). The contextual determinants of presidential greatness. Presidential Studies Quarterly, 18, 161–169.
Klecka, W. R. (1980). Discriminant Analysis (Sage University Paper Series on Quantitative Applications in the Social Sciences, series no. 07–044). Beverly Hills, CA: Sage.
Kynerd, T. (1971). An analysis of presidential greatness and “presidential rating”. Southern Quarterly, 9, 309–329.
LeBreton, J. M., & Senter, J. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11, 815–852.
Ligon, G. S., & Hunter, S. T. (2010). Putting the development into experiential development. Industrial and Organizational Psychology, 3, 28–32.
Ligon, G. S., Hunter, S. T., & Mumford, M. D. (2008). Development of outstanding leadership: A life narrative approach. The Leadership Quarterly, 19, 312–334,
http://dx.doi.org/10.1016/j.leaqua.2008.03.005.
Ligon, G. S., Leahy, Z., Versella, M., Troyan, C., Gibson, P., Hunter, S. T., et al. (2010). Structure differences between violent and non-violent ideological
organizations. Poster presented at the 2010 Society of Industrial and Organizational Psychology Annual Conference, Atlanta, GA.
Lilienfeld, S. O., Waldman, I. D., Landfield, K., Watts, A. L., Rubenzer, S., & Faschingbauer, T. R. (2012). Fearless dominance and the U.S. presidency: Implications of
psychopathic personality traits for successful and unsuccessful political leadership. Journal of Personality and Social Psychology, http://dx.doi.org/10.1037/a0029392
(Advance online publication).
Marmor-Lavie, G., & Weimann, G. (2005). Measuring emotional appeals in Israeli election campaigns. International Journal of Public Opinion Research, 18, 318–339.
McAdams, D. P. (2006). The redemptive self: Stories Americans live by. New York, NY: Oxford University Press.
McCall, M. W. (2010). Recasting leadership development. Industrial and Organizational Psychologist, 3, 3–21.
McCann, S. J. H. (1992). Alternative formulas to predict the greatness of U.S. Presidents: Personological, situational, and zeitgeist factors. Journal of Personality and
Social Psychology, 62, 469–479.
McCann, S. H. (2005). Longevity, Big Five personality factors, and health behaviors: Presidents from Washington to Nixon. Journal of Psychology, 139, 273–286.
McCullough, D. (2001). John Adams. NY, NY: Simonton & Schuster.
McKelvey, B., Mintzberg, H., Petzinger, T., Prusak, L., Senge, P., & Shultz, R. (1999). The gurus speak: Complexity and organizations. Emergence: A Journal of
Complexity Issues in Organizations and Management, 1, 73–91.
McKeown, B. F., & Thomas, D. (1988). Q methodology: Sage University Paper series on Quantitative Applications in the Social Sciences, No. 66–001. Beverly Hills and
London: Sage Publications.
Miller, N. L., & Stiles, W. B. (1986). Verbal familiarity in American nomination acceptance speeches and inaugural addresses (1920–1981). Social Psychology
Quarterly, 49, 72–81.
Mills, S. (2003). Michel Foucault. New York: Routledge.
Moghal, M. (2010). Idi Amin: Lion of Africa. Central Milton Keynes: Author House.
Mosely, L. (1985). Disney's World. New York: Stein and Day.
Mumford, M. D. (2006). Pathways to outstanding leadership: A comparative analysis of charismatic, ideological, and pragmatic leaders. New Jersey: Lawrence Erlbaum
Associates, Inc.
Mumford, M. D., Bedell, K. E., Hunter, S. T., Espejo, J., & Boatman, P. R. (2006). Problem-solving. In M. D. Mumford (Ed.), Pathways to outstanding leadership:
A comparative analysis of charismatic, ideological, and pragmatic leadership. Mahwah, NJ: Erlbaum Press.
Mumford, M. D., Connelly, M., Scott, G., Espejo, J., Sohl, L. M., Hunter, S. T., et al. (2005). Career experiences and scientific performance: A study of social, physical,
life, and health sciences. Creativity Research Journal, 17, 105–129, http://dx.doi.org/10.1207/s15326934crj1702&3_1.
Mumford, M. D., Espejo, J., Hunter, S. T., Bedell-Avers, K. E., Eubanks, D. L., & Connelly, S. (2007). The sources of leader violence: A comparison of ideological and
non-ideological leaders. The Leadership Quarterly, 18, 217–235, http://dx.doi.org/10.1016/j.leaqua.2007.03.005.
Mumford, M. D., Gaddis, B., Licuanan, B., Ersland, B., & Siekel, K. (2006). Communication strategies. In M. D. Mumford (Ed.), Pathways to outstanding leadership:
A comparative analysis of charismatic, ideological, and pragmatic leadership. Mahwah, NJ: Erlbaum Press.
Mumford, M. D., Gaddis, B., Strange, J. M., & Scott, G. (2006). General method. In M. D. Mumford (Ed.), Pathways to outstanding leadership: A comparative analysis of
charismatic, ideological, and pragmatic leadership. Mahwah, NJ: Erlbaum Press.
Mumford, M. D., Scott, G., & Hunter, S. T. (2006). Theory. In M. D. Mumford (Ed.), Pathways to outstanding leadership: A comparative analysis of charismatic,
ideological, and pragmatic leadership. Mahwah, NJ: Erlbaum Press.
Mumford, M. D., Stokes, G. S., & Owens, W. A. (1990). Patterns of life history: The ecology of human individuality. Hillsdale, NJ England: Lawrence Erlbaum
Associates, Inc.
Mumford, M. D., Strange, J. M., Gaddis, B., Licuanan, B., & Scott, G. (2006). Performance. In M. D. Mumford (Ed.), Pathways to outstanding leadership: A comparative
analysis of charismatic, ideological, and pragmatic leadership. Mahwah, NJ: Erlbaum Press.
Mumford, M. D., & Threlfall, V. (1992). Quantifying genius. Psyccritiques, 37, 216–218, http://dx.doi.org/10.1037/031362.
Mumford, M. D., & Van Doorn, J. R. (2001). The leadership of pragmatism: Reconsidering Franklin in the age of charisma. The Leadership Quarterly, 12, 279–309,
http://dx.doi.org/10.1016/S1048-9843(01)00080-7.
Murray, E. (1928). An historiometric study of the early traits of great orators. The Quarterly Journal of Speech, 14, 502–508.
National Consortium for the Study of Terrorism and Responses to Terrorism (START) (2012). Global Terrorism Database. Retrieved from. http://www.start.umd.
edu/gtd
Neal, M. W., & Tansey, R. (2010). The dynamics of effective corrupt leadership: Lessons from Rafik Hariri's political career in Lebanon. The Leadership Quarterly, 21, 33–49.
O'Connor, J., Mumford, M. D., Clifton, T. C., Gessner, T. L., & Connelly, M. (1995). Charismatic leaders and destructiveness: An historiometric study. The Leadership
Quarterly, 6, 529–555, http://dx.doi.org/10.1016/1048-9843(95)90026-8.
Osterlind, S. J. (1998). Constructing test items: Multiple-choice, constructed-response, performance, and other formats (2nd edition). Boston, MA: Kluwer.
Panagopoulos, C. (2004). Boy talk/girl talk: Gender differences in campaign communication strategies. Women & Politics, 26, 131–155.
Perret, G. (2002). Jack: A life like no other. New York: Random House, Inc.
Pillemer, D. B. (2001). Momentous events and the life story. Review of General Psychology, 5, 123–134.
Rubenzer, S. J., Faschingbauer, T. R., & Ones, D. S. (2000). Assessing the U.S. presidents using the revised NEO Personality Inventory. Assessment, 7, 403–420,
http://dx.doi.org/10.1177/107319110000700408.
Sageman, M. (2008). Leaderless Jihad. Philadelphia, Pennsylvania: University of Pennsylvania Press.
G.S. Ligon et al. / The Leadership Quarterly 23 (2012) 1104–1133
1133
Scheuer, M. (2011). Osama bin Laden. New York, New York: Oxford University Press, Inc.
Schriesheim, C. A., Castro, S. L., & Cogliser, C. C. (1999). Leader–member exchange (LMX) research: A comprehensive review of theory, measurement, and
data-analytic practices. The Leadership Quarterly, 10, 63–114.
Shawcross, W. (1997). Murdoch. New York, NY: Touchstone.
Simonton, D. K. (1975). Sociocultural context of individual creativity: A transhistorical time series analysis. Journal of Personality and Social Psychology, 32,
1119–1133, http://dx.doi.org/10.1037/0022-3514.32.6.1119.
Simonton, D. K. (1976). Biographical determinants of achieved eminence: A multivariate approach to the Cox data. Journal of Personality and Social Psychology, 33,
218–226, http://dx.doi.org/10.1037/0022-3514.33.2.218.
Simonton, D. K. (1980). Land battles, generals, and armies: Individual and situational determinants of victory and casualties. Journal of Personality and Social
Psychology, 38, 110–119.
Simonton, D. K. (1981). Presidential greatness and performance: Can we predict leadership in the White House? Journal of Personality, 49, 306–323.
Simonton, D. K. (1983). Intergenerational transfer of individual differences in hereditary monarchs: Genetic, role-modeling, cohort, or sociocultural effects?
Journal of Personality and Social Psychology, 44, 354–364, http://dx.doi.org/10.1037/0022-3514.44.2.354.
Simonton, D. K. (1984). Leaders as eponyms: Individual and situational determinants of monarchal eminence. Journal of Personality, 52, 1–21.
Simonton, D. K. (1985). The vice-presidential succession effect: Individual or situational basis? Political Behavior, 7, 79–99.
Simonton, D. K. (1986a). Presidential personality: Biographical use of the Gough Adjective Checklist. Journal of Personality and Social Psychology, 51, 149–160.
Simonton, D. K. (1986b). Dispositional attributions of (presidential) leadership: An experimental simulation of historiometric results. Journal of Experimental
Social Psychology, 22, 389–418, http://dx.doi.org/10.1016/0022-1031(86)90042-9.
Simonton, D. K. (1987). Presidential inflexibility and veto behavior: Two individual-situational interactions. Journal of Personality, 55, 1–18.
Simonton, D. K. (1988a). Presidential style: Personality, biography, and performance. Journal of Personality and Social Psychology, 55, 928–936, http://dx.doi.org/
10.1037/0022-3514.55.6.928.
Simonton, D. K. (1988b). Quality and purpose, quantity and chance. Creativity Research Journal, 1, 68–74, http://dx.doi.org/10.1080/10400418809534289.
Simonton, D. K. (1988c). Galtonian genius, Kroeberian configurations, and emulation: A generational time-series analysis of Chinese civilization. Journal of
Personality and Social Psychology, 55, 230–238, http://dx.doi.org/10.1037/0022-3514.55.2.230.
Simonton, D. K. (1990). Creativity and wisdom in aging. In J. E. Birren, K. Schaie, J. E. Birren, & K. Schaie (Eds.), Handbook of the psychology of aging (pp. 320–329).
(3rd ed.). San Diego, CA US: Academic Press.
Simonton, D. K. (1991). Latent-variable models of posthumous reputation: A quest for Galton's G. Journal of Personality and Social Psychology, 60, 607–619.
Simonton, D. K. (1992). Gender and genius in Japan: Feminine eminence in masculine culture. Sex Roles, 27, 101–119.
Simonton, D. K. (1997). Foreign influence and national achievement: The impact of open milieus on Japanese civilization. Journal of Personality and Social
Psychology, 72, 86–94, http://dx.doi.org/10.1037/0022-3514.72.1.86.
Simonton, D. K. (1998a). Historiometric methods in social psychology. European Review of Social Psychology, 9, 267–293.
Simonton, D. K. (1998b). Mad King George: The impact of personal and political stress on mental and physical health. Journal of Personality, 66, 443–466.
Simonton, D. K. (2001). Predicting presidential performance in the United States: Equation replication on recent survey results. Journal of Social Psychology, 141,
293–307.
Simonton, D. K. (2003). Qualitative and quantitative analyses of historical data. Annual Review of Psychology, 54, 617–640.
Simonton, D. K. (2006). Presidential IQ, openness, intellectual brilliance, and leadership: Estimates and correlations for 42 U.S. Chief Executives. Political
Psychology, 27, 511–526, http://dx.doi.org/10.1111/j.1467-9221.2006.00524.x.
Simonton, D. K. (2008). Childhood giftedness and adulthood genius: A historiometric analysis of 291 eminent African Americans. Gifted Child Quarterly, 52, 243–255.
Simonton, D. K. (2010). Emotion and composition in classical music: Historiometric perspectives. In P. N. Juslin, J. A. Sloboda, P. N. Juslin, & J. A. Sloboda (Eds.),
Handbook of music and emotion: Theory, research, applications (pp. 347–366). New York, NY US: Oxford University Press.
Spangler, W. D., & House, R. J. (1991). Presidential effectiveness and the leadership motive profile. Journal of Personality and Social Psychology, 60, 439–455,
http://dx.doi.org/10.1037/0022-3514.60.3.439.
Stephenson, W. (1935). Technique of factor analysis. Nature, 136, 295–305.
Strange, J. M. (2004). Exceptional leaders and their followers: An in-depth look at leader follower interactions. Unpublished Dissertation.
Strange, J. M., & Mumford, M. D. (2002). The origins of vision: Charismatic versus ideological leadership. The Leadership Quarterly, 13, 343–377, http://dx.doi.org/
10.1016/S1048- 9843(02)00125-X.
Suedfeld, P., & Bluck, S. (1988). Changes in integrative complexity prior to surprise attacks. Journal of Conflict Resolution, 32, 626–635.
Suedfeld, P., & Rank, A. (1976). Revolutionary leaders: Long-term success as a function of changes in conceptual complexity. Journal of Personality and Social
Psychology, 34, 169–178, http://dx.doi.org/10.1037/0022-3514.34.2.169.
Suedfeld, P., Tetlock, P. E., & Ramirez, C. (1977). War, peace and integrative complexity: UN speeches on the Middle East problem, 1947–1976. Journal of Conflict
Resolution, 31, 123–132.
Szulc, R. (1986). Fidel: A critical portrait. New York: William Morrow and Company.
Tetlock, P. E. (1979). Identifying victims of groupthink from public statements of decision makers. Journal of Personality and Social Psychology, 37, 1313–1324.
Tetlock, P. E. (1981). Personality and isolationism: Content analysis of senatorial speeches. Journal of Personality and Social Psychology, 41, 737–743.
Tetlock, P. E. (1983). Cognitive style and political ideology. Journal of Personality and Social Psychology, 45, 118–126.
Thoemmes, F. J., & Conway, L. (2007). Integrative complexity of 41 U.S. presidents. Political Psychology, 28, 193–226, http://dx.doi.org/10.1111/j.1467-9221.2007.00562.x.
Thorndike, E. (1936). Relation between intellect and morality in rulers. The American Journal of Sociology, 42, 321–334.
Traub, R. E. (1994). Reliability for the Social Sciences: Theory and Applications, Vol. 3, Thousand Oaks, CA: Sage.
Wallace, M. D., Suedfeld, P., & Thachuk, K. L. (1993). Political rhetoric of leaders under stress: Findings from the Gulf crisis. Journal of Conflict Resolution, 37,
94–107.
Watson, G. B. (1927). Post mortem mental tests for eminent men. Journal of Educational Psychology, 18, 202–206, http://dx.doi.org/10.1037/h0065262.
Weber, M. (1926). The theory of social and economic organizations. New York: Free Press.
Wendt, H. W., & Muncy, C. A. (1979). Studies of political character: Factor patterns of 24 U.S. vice-presidents. Journal of Psychology, 102, 125–131.
Winter, D. G. (1980). Measuring the motives of southern Africa political leaders at a distance. Political Psychology, 2, 75–85.
Winter, D. G. (1987). Leader appeal, leader performance, and the motive profiles of leaders and followers: A study of American presidents and elections. Journal of
Personality and Social Psychology, 52, 196–202.
Woods, F. A. (1909). A new name for a new science. Science, 30, 703–704.
Woods, F. A. (1911). Historiometry as an exact science. Science, 33, 568–574.
Yammarino, F. J., Mumford, M. D., Serban, A., & Shirreffs, K. (2012). Assassination and leadership. Unpublished manuscript.
Zullow, H. M., & Seligman, M. P. (1990). Pessimistic rumination predicts defeat of presidential candidates, 1900 to 1984. Psychological Inquiry, 1, 52–62.