Proceedings of the 38th Hawaii International Conference on System Sciences - 2005
A Meta-Analysis of Role Ambiguity and Role Conflict on IS Professional Job
Satisfaction
Yide Shen
CIS Department
Robinson College of Business
Georgia State University
Atlanta, GA, USA 30302
[email protected]
Abstract
There have been numerous IS studies on the topic of role
ambiguity (RA) and role conflict (RC). The need for
resolving the disagreement in understanding the impact of
RA and RC on job satisfaction within an IS context
motivated the current research. Employing Hunter and
Schmidt’s (1990) meta-analysis method, this study
synthesizes empirical IS study results from twelve
independent samples. The results revealed that both RA
and RC are negatively correlated with IS professional’s
job satisfaction across diverse studies. Compared with
non-IS related jobs, IS professionals are not necessarily
more- or less-susceptible to the negative impacts of RA
and RC on job satisfaction. Furthermore, the unexplained
variance in the study effect sizes indicates the existence of
potential moderating variables. Finally, an unexpected
but noteworthy contribution of the current research is to
raise the issue of misuse of the term meta-analysis within
the IS research community.
1. Introduction
With the proliferation of computers and information
systems in organizations, information technology (IT)
professionals are playing increasingly critical roles in
supporting organizational operations and attaining
strategic goals. Hence, understanding the factors that
influence IS professionals’ job satisfaction is an area
which concerns both IS managers and IS researchers. The
importance of this issue is reinforced by research showing
that job satisfaction is the most important antecedent of IS
employees’ intention to stay in an organization and thus,
the most important determinant of IS personnel turnover
[3]. Additionally, decreased job satisfaction often leads to
increased absenteeism, ill health, and grievances [40],
which impairs the quality of work life of IS professionals.
Among the variables expected to influence job
satisfaction, two role stressors – role ambiguity (RA) and
role conflict (RC) – have captured the attention of
numerous researchers. The results from both the
organizational behavior and IS literatures suggest that RA
and RC are generally negatively related to job
satisfaction, however the strength and significance level
of these relationships are not always consistent.
Especially in IS research, we lack a definitive conclusion
about the relative importance of RA and RC.
The need for resolving the disagreement in
understanding the impact of RA and RC on job
satisfaction within an IS context motivated the current
research. Given the broad range of empirical IS research
on this topic, a formal meta-analysis is useful to
accumulate results across many independent studies and
synthesize the statistical data from multiple sources. In
addition, due to the uniqueness and rapid changes
occurring in the IT environment, it is worthwhile to
empirically examine the applicability of role stressor-job
satisfaction relationships in IS settings. This is the
objective of this paper: to accumulate empirical results
regarding the relationship between RA/RC and IS
professionals’ job satisfaction across independent studies,
to examine the relative importance of the two role
stressors, to compare them with those from a broad range
of job occupations in the organizational behavior
literature, and investigate the variables that may moderate
these relationships.
The remainder of the paper is structured as follows.
Section 2 reviews the concepts of RA and RC as they
have been described in the management literature and
develops hypotheses for this study. Sections 3 and 4
describe the meta-analysis method employed in this study.
Section 5 presents the result of our analysis. The paper
concludes by discussing the results from the quantitative
analysis and their implications for IS personnel
researchers and IS scholars, more broadly.
2. Literature Review and Hypotheses
“A role is most typically defined as a set of
expectations about behavior for a position in a social
structure” [47, p. 155]. RA refers to the degree to which a
given job is lacking in terms of 1) the predictability of
0-7695-2268-8/05/$20.00 (C) 2005 IEEE
1
Proceedings of the 38th Hawaii International Conference on System Sciences - 2005
responses to one’s behavior, and 2) the existence or
clarity of behavior requirements [47]. RC is defined as
“the degree of incongruity or incompatibility in the
expectations or requirements communicated to a focal
person” [4, p. 93]. It is generally believed that RA and RC
result in lower job satisfaction. Consistent with the
theoretical basis, most of the empirical results reported
medium to high negative correlations between either RA
or RC and job satisfaction, while only two studies have
shown positive correlations [25, 32]. In line with the
goals of this research – to accumulate empirical results
regarding the relationship between RA/RC and IS
professionals’ job satisfaction across independent studies
– we propose the first two hypotheses:
Hypothesis 1: RA and job satisfaction will be
negatively correlated among IS professionals.
Hypothesis 2: RC and job satisfaction will be
negatively correlated among IS professionals.
Another inconsistency that has been observed is the
relative impact of either RA or RC on job satisfaction. For
instance, by studying a sample of 229 IS professionals
from nine companies in various industries (e.g.,
insurance, banking, manufacturing, etc.), Baroudi [2]
found that RA is a more influential variable, accounting
for 22% of the variance in job satisfaction, compared to
RC, which accounted for only 7% of the variance in job
satisfaction. In contrast, based on their study of 109 IS
managers from a wide variety of industries (e.g., banking,
insurance, manufacturing, etc.), Li and Shani [37]
concluded that:
Role ambiguity, having the lowest mean composite score
[2.9] among the four stressors perceived by the IS
managers [work overload (5.0), role conflict (4.5), jobinduced anxiety (3.2), and role ambiguity (2.9)], is
significantly influenced by many organizational
contextual as well as job satisfaction factors. Ivancevich,
Napier, and Wetherbe (1983) had a similar finding that
role ambiguity was not a major stressor perceived by the
IS personnel; it was the fifth highest mean score among
the seven stressor scales used in their study [37, p. 123].
However, a closer review of these two studies reveals
some differences in terms of the research design. First,
Baroudi’s sample consisted of IS professionals from a
range of job types (such as applications programmers,
programmer/analysts, analysts, and project leaders), while
the subjects in Li and Shani’s sample were all IS
managers. Second, the two studies used the same measure
for RA and RC [47], but they used different measures for
job satisfaction (Job Descriptive Index [50] vs. a homegrown scale [16]). Thus, it is impossible to conclude
whether the different impacts of RA and RC on job
satisfaction are due to the nature of these two constructs,
or whether the contradictory results are introduced by
these other artifacts.
Based on above discussion, hypothesis 3 compares the
magnitude of the negative effects of RA and RC on job
satisfaction:
Hypothesis 3: The correlation between RA and job
satisfaction will differ from the correlation between
RC and job satisfaction among IS professionals.
Two meta-analysis studies from the organizational
behavior literature have identified a generally negative
relationship between both RA and RC and job satisfaction
in many different types of occupations, although the
strength of the relationships varied across different
occupational samples [10, 28]. Kahn and colleagues’ [30]
study of various jobs revealed that the level of RC and
RA were high among people whose jobs involved
innovation or boundary spanning (i.e., coping with people
outside the person’s work group).
This is exactly the nature of most types of job in the IS
work environment. For example, rapid changes in
technological innovation require IS employees to update
their knowledge and skills frequently. System analysts
need to work with cross- departmental and even crossorganizational stakeholders. These characteristics indicate
that the problem of high RC and RA especially pertains to
IS professionals:
The evidence is highly suggestive, therefore, that systems
professional with their high … need for achievement,
autonomy, and cognition, and low need for change are
particularly prone to the adverse effects of role conflict
and role ambiguity [2, p. 342].
On the other hand, it is possible that because role
stressors are inherent within IS work environments,
workers that have been selected for and socialized to
work within the IS field have a relatively high tolerance
for RA and RC [23]. King and Sethi [31] described how
the process of socialization may make IS professionals
accustomed to accepting such challenges in their jobs.
Thus, even though the levels of RA and RC may be high
for IS professionals, their high levels of tolerance for role
ambiguity and role conflict may help to moderate the
negative effects that would otherwise occur, in terms of
low job satisfaction.
Based on above discussion, hypothesis 4 compares the
negative effects of RA and RC between IS professionals
and other non-IS job types:
Hypothesis 4a: The strength of the relationship
between RA and job satisfaction will be similar among
IS professionals and other job types.
Hypothesis 4b: The strength of the relationship
between RC and job satisfaction will be similar among
IS professionals and other job types.
0-7695-2268-8/05/$20.00 (C) 2005 IEEE
2
Proceedings of the 38th Hawaii International Conference on System Sciences - 2005
3.2. Coding of Study Information
3. Meta-Analysis Method
The term meta-analysis was coined by Glass [11, p. 3]
to refer to “the analysis of analyses…the statistical
analysis of a large collection of analysis results from
individual studies for the purpose of integrating the
findings.” The main thrust of meta-analysis methods is to
accumulate empirical results across independent studies
and provide a more accurate representation of population
characteristics. Over the years, meta-analysis has become
a legitimate statistical tool to integrate empirical research
findings in many disciplines, such as medicine, education,
and psychology [19].
Hunter and Schmidt’s [18] psychometric meta-analysis
is the primary method employed in this study. The focus
of Hunter and Schmidt’s method is on effect sizes, which
represent correlational measures of effect. The advantage
of Hunter and Schmidt’s approach is that it provides
techniques to correct for artifactual and methodological
problems that may distort effect sizes from the true score
values. In survey research, correlations (Pearson’s r) are
the most frequently-used measure of effect size, and the
focus of the current research. (In contrast, controlled
laboratory studies employ other types of effect sizes to
characterize the differences between experimental and
control group effects, such as Cohen’s d.) Following the
processes suggested by Hunter and Schmidt [18], our
meta-analysis consists of three major phases: literature
search, coding of study information (both methods and
results), and data analysis. Each phase is discussed in
detail in the following sections.
3.1. Literature Search
A literature research was performed primarily on
electronic sources (ABI/Inform, EBSCO Business Source
Premier, ACM Digital Library, and ScienceDirect), as
well as several conference proceedings (ICIS, HICSS,
AMCIS, and ACM SIG CPR) using the keywords “role
ambiguity”, “role conflict”, “job satisfaction”, “stress”,
“role stress,” “burnout”, and “work exhaustion”. Since the
focus of this research is on IS professionals, the term
“information systems” and “information technology”
were used in combination with the above keywords to
restrict the search results. We also reviewed the reference
list of the articles retrieved to identify other related
papers. The search yielded 40 papers related to RA, RC,
and job satisfaction in an IS context. Methodological
details and correlational results were then coded based on
these papers.
A total of 25 papers were excluded from preliminary
screening for various reasons. Since the focus of this
study is on IS professionals, four papers examining end
users or other non-IS occupational types were excluded.
Meta-analysis requires data regarding correlations and
sample sizes in order to compute a weighted correlation
between independent and dependent variables, thus,
another eighteen papers were excluded because they
either were theoretical papers without any empirical data
(two papers) or didn’t report correlations for either of the
role stressor variables and job satisfaction (sixteen
papers). In addition, three other papers that did not study
job satisfaction were also excluded.
Following Viswesvaran and Ones’s [51] advice,
methodological weaknesses were not treated as grounds
to eliminate any studies. For example, if a paper reported
a weak result for internal reliability (Cronbach alpha) for
RA or RC, rather than excluding the study, we
incorporated the information about the (weak) reliability
metrics, and used Hunter and Schmidt’s procedures to
correct for this methodological weakness (and others).
The remaining fifteen papers were then coded for the
following information: sample size, Pearson’s correlations
between the role stressors and job satisfaction, and
reliability statistics (Cronbach’s alpha). During the coding
process, we found that five data samples were each used
in two different papers, respectively. To avoid double
counting of the same sample, only one paper for each
sample was included in our analysis.
Furthermore, two papers separated their sample into
two sub-groups. One of these studies [25] split the sample
randomly, using one sub-sample to validate their
instruments and another sub-sample to test their research
model. The two subgroups from this study were thus
treated as independent samples in our analysis. The other
study [26] divided the sample according to the mean
value of a specific personality variable (Type A versus
Type B personality behavior), and found a moderating
effect of this variable. Hunter and Schmidt [18] suggested
using subgroup correlations for the two subgroups
separately if there is evidence of a moderating effect. The
two subgroups from the latter study were also treated as
independent samples. This left us with a final sample size
of ten papers consisting of twelve independent samples
(Table 1). Of the twelve samples, all of them reported
correlations between RA and job satisfaction, but only ten
of the twelve samples reported correlations between RC
and job satisfaction. The meta-analysis for each set of
variables is discussed in the following section.
0-7695-2268-8/05/$20.00 (C) 2005 IEEE
3
Proceedings of the 38th Hawaii International Conference on System Sciences - 2005
Table 1. Final sample for meta-analysis
r
Study
Sample # Sample N
1 Goldstein Rockart 1984
2 Bostrom 1981
Ivancevich, Napier, and
3 Wetherbe 1985
4 Li Shani 1991
Igbaria, Parasuraman, and
5 Badawy 1994
Cronbach's Alpha
Role
Role
Job
RA Æ JS RC Æ JS
Ambiguity Conflict
Satisfaction
-0.57
-0.40
0.81
0.79
0.80
1
118
2
3
75
221
-0.45
-0.38
-0.34
~
4
225
-0.02
~
~
~
~
0.85
~
0.93
5
109
-0.63
-0.41
0.91
0.85
0.84
6
224
0.39
0.43
0.86
0.90
0.86
7
214
0.54
0.46
0.88
0.89
0.88
6 Guimaraes Igbaria 1992
7 Igbaria Chidambaram 1997
8
209
-0.46
-0.36
0.84
0.78
0.82
9
348
-0.50
-0.41
0.82
0.87
0.70
8 Baroudi 1985
9 Lee 2000
10
229
-0.51
-0.39
0.85
0.79
0.91
11
273
-0.44
-0.17
0.81
0.82
0.83
10 King and Xia 2001
12
187
0.15
-0.50
0.71
0.70
0.86
* RA Æ JS: correlation between RA and Job Satisfaction;
RC Æ JS: correlation between RC and Job Satisfaction.
4. Data Analysis
There are four steps to conducting a meta-analysis:
correcting for any study artifacts, calculating effect sizes,
computing the Fail-Safe N, and evaluating the homogeneity or heterogeneity of sample effects.
The first step is to correct for measurement error. With
the reliability statistics for both independent and
dependent variables, measurement error was corrected
using the formula provided by Hunter and Schmidt [18].
In cases where the reliability values were missing in the
samples statistics, the averaged Cronbach’s alpha value
was used instead. After correcting for measurement error,
the sampling error was also corrected. The second step is
to calculate the effect size by averaging the reported
correlations, weighted by sample size [18, p. 150]. The
formula for effect size is
r=
¦ [N r ] , where r is the
¦N
i i
i
i
correlation after correcting for measurement error in study
i and Ni is the number of subjects in study i.
The third step is to calculate the Fail-Safe N, which is
a way to address the “file-drawer” problem – the
sampling bias in meta-analysis which assumes that an
unknown number of studies with effect sizes of zero were
not included in the analysis because they have either not
been submitted for publication or have been rejected [48].
That is to say, the actual mean effect size may not be as
high as the one generated by meta-analysis, due to the
fact any studies reporting weak or zero effect sizes are
less likely to be published, and thus less likely to be
included in the meta-analysis. However, “the combined
study results can be highly significant statistically even
though the mean effect size is small or even tiny” [18, p.
512]. Hunter and Schmidt [18] further provided the
formula to calculate Fail-Safe N, which indicates the
number of missing studies with zero effect size that would
have to exist to bring the mean effect size down to some
specific level (0.1 in this study). The lower the value of
the “Fail-Safe N” (e.g., a value = 2), then the more
uncertainty that exists with regard to whether the mean
effect size may be biased by the number of studies are
“buried in the file drawer”, due to weak, non-existent, or
contrary effects.
The fourth step examines whether the results are
consistent (homogeneous) across studies, or whether there
are large variations in the pattern of correlations. The
meta-analysis results are meaningful only if the various
samples are homogeneous. The purpose of the
homogeneity test is then to inspect whether any variations
in results across studies are due to sampling error, or due
to possible moderator variables [18]. Two criteria were
applied to evaluate homogeneity of the sample –
unexplained variance and the Q test of homogeneity. As a
rule of thumb, Hunter and Schmidt [18] suggested that if
the fraction of unexplained variance is less than 25%, this
variance can be neglected (i.e., due to some unmeasurable
errors). Otherwise, if the percentage of unexplained
variance is greater than 25%, there might be potential
moderators that affect the correlation. The second
criterion is the Q test of homogeneity. If the Q test value
is not significant, then there is no true variation across
studies; but if it is significant, it is reasonable to conclude
that there are moderating variables influencing the
correlations found from various studies.
0-7695-2268-8/05/$20.00 (C) 2005 IEEE
4
Proceedings of the 38th Hawaii International Conference on System Sciences - 2005
Table 2. Results of meta-analysis
Correlate
k
Total N
Mean Effect
Size
95% C.I.
Lower
Higher
% Unexplained
Fail-Safe N Variance in sample Homogeneity Test
correlations
Q
p
RA* --> JS
12
2432
-0.271
-0.311
-0.232
21
97.96%
502.737
0.0000
RC* --> JS
10
1986
-0.243
-0.287
-0.199
15
97.39%
339.646
0.0000
* RA Æ JS: correlation between RA and Job Satisfaction;
RC Æ JS: correlation between RC and Job Satisfaction.
Safe N values (21 and 15 for RA and RC, respectively)
are a sign of considerably strong results.
5. Results and Discussion
5.1. Hypotheses Testing
5.1.1. Hypothesis 1 and 2 – Supported. The results of
the meta-analysis are summarized in Table 2. Table 2
shows that the mean effect size for the correlation of RA
and job satisfaction is -0.271, and that between RC and
job satisfaction is -0.243. In general, both role stressor
variables – RA and RC – are negatively correlated with
job satisfaction. Thus, hypotheses 1 and 2 are both
supported. Based on Cohen’s [7] standard effect size
levels of 0.10, 0.30, and 0.50 as small, medium, and large,
respectively, the strength of the relationship between RA
and job satisfaction (-0.271) is close to medium, while
that between RC and job satisfaction (-0.243) is
somewhat weaker than medium.
Another principle to judge the significance of the mean
effect size is based on confidence intervals. According to
[39], if the confidence interval does not include the value
of zero, then the mean effect size is significant at the level
specified by the confidence intervals. The confidence
interval shown in Table 2 indicate that both of the mean
effect sizes are significant at the p<0.05 level, because
zero is not included in the confidence intervals.
Furthermore, according to Hunter and Schmidt [18],
even when the computed mean effect size is relatively
small, the cumulated study results can still be statistically
significant. When we examine the Fail-Safe N for each
relationship, we would need to locate 21 additional
studies with zero-effect sizes in order to reduce the
calculated mean correlation of 0.271 between RA and job
satisfaction to 0.1 (the threshold for weak correlation)
when our result is calculated based on the 12 studies that
we did find and code. Moreover, 15 studies with zeroeffect sizes would be required to reduce the mean effect
size from 0.243 correlation across studies to 0.1 when we
calculated the mean correlation between RC and job
satisfaction based on 10 studies.
In summary, both RA and RC are negatively correlated
with job satisfaction with medium strength (for RA) and
somewhat less than medium strength (for RC). The
correlations are significant at 0.05 level. The large Fail-
5.1.2. Hypothesis 3 – Not Supported. We hypothesized
that the magnitude of the negative effects of RA and RC
on job satisfaction would be different. To test this
hypothesis, the non-parametric Wilcoxon paired signed
rank test was conducted based on the corrected correlation
coefficient multiplied by the sample sizes for each
individual study. The result indicates that the correlations
of RA and RC with job satisfaction (after correcting for
sampling error and measurement error), did not differ
significantly from each other (Z=-1.078, p=0.139). Thus,
RA and RC have a statistically indistinguishable negative
effect on job satisfaction. Hypothesis 3 is disconfirmed.
5.1.3. Hypothesis 4 – Supported. The two meta-analysis
studies [10, 28] from the organizational behavior
literature provide a comparative basis for Hypothesis 4.
Jackson and Schuler’s [28] meta-analysis study included
96 papers none of which featured IS professionals. Fisher
and Gitelson’s [10] study reviewed 43 studies and 59
independent samples. Even though they didn’t provide
population information for the samples, a review of the
studies’ titles included in their bibliography did not
indicate the presence of any IS-related samples.
Therefore, the synthesized effect sizes from these two
studies represent the relationship between RA/RC and job
satisfaction across various types of occupations (which
appear not to have included IS employees).
The mean effect sizes in the current study are, by and
large, comparable to these two prior meta-analysis
studies. More specifically, the correlation between RA
and job satisfaction from this study (-0.271) falls between
the effect sizes obtained from the two prior meta-analyses
published in the organizational behavior literature (-0.25
and -0.30, respectively). In contrast, the correlation
between RC and job satisfaction in our study (-0.243) is
weaker than the effect sizes from both of these prior
studies (-0.35 and -0.31, respectively).
To further test the statistical significance of above
comparisons, a non-parametric Mann-Whitney U test was
0-7695-2268-8/05/$20.00 (C) 2005 IEEE
5
Proceedings of the 38th Hawaii International Conference on System Sciences - 2005
performed. Since only one of the organizational behavior
studies [28] provided the original correlations for each
individual study in its meta-analysis, the correlations from
the current study are compared only with those from
Jackson and Schuler [28].1
The results of the Mann-Whitney U test indicated that
the correlations between RA and job satisfaction among
IS professionals are not statistically different from other
job types (Z = -0.049, p=0.961). Similarly, correlations
between RC and job satisfaction did not differ statistically
between IS professionals and other job types (Z = -0.895,
p=0.371). Thus, both Hypothesis 4a and 4b are supported.
While the mean effect size for the correlation between RC
and job satisfaction was slightly weaker (-0.243),
compared to the two prior meta-analyses for other job
types (-0.35 and -0.31)., this difference was not
statistically significant. .
These results indicate that even though most IS jobs
require both innovation and boundary spanning [2],
which, in turn, introduce highs level of RA and RC,
respectively [30], the negative impacts of RA and RC are
not necessarily stronger among IS professionals than for
other occupations. One potential explanation of this
phenomenon is that IS professionals are selected for (or
have learned to have) high tolerance for ambiguity and
role conflict [23, 31].
5.2. Moderating Effect
The unexplained variances in both of the correlations
are higher than the recommended threshold of 25%. This
indicates that sampling error only accounts for a small
percentage of the variance in sample correlations (2.92%
for RA, 3.51% for RC). Thus, there may be one or more
moderators having influence on the correlations found
from various studies. The highly significant homogeneity
Q test results also corroborates this conclusion. It is also
consistent with that of the two organizational behavior
meta-analyses [10, 28], namely, theoretically meaningful
moderators should be included to explain the variance of
both correlations across the samples.
On the other hand, the unexplained variance and the
significant Q test result from this study are both even
higher than the results obtained from general job settings.
However, considering that we conducted this analysis
only among IS professional and excluded the samples of
IT end users , we have , to some extent, controlled for
variations in job type. The significant results from the
homogeneity test suggest that large variations among the
broad category of IS professionals may exist. This may
indicate that a further specification of the job types of IS
1
In addition, because the reliability statistics (Cronbach’s
alpha) for each individual study are not shown in Jackson and
Schuler’s (1985) paper, our analysis is based on the original
correlations, instead of the weighted correlations after correcting
for sampling error and measurement error.
professionals are required, rather than simply aggregating
all employees together under the general category of IS
professionals.
According to Hunter and Schmidt [18], subgroup
analysis can be performed to detect possible moderator
variables. The hypothesized moderator variable can be
used to split the sample into two or more subsamples. The
same meta-analysis procedures described above then can
be carried out for each subset independently. For
example, the data could be analyzed separately for all
samples that consisted of “systems analysts,” and another
analysis for samples consisting of “project managers”. If
the variance of the effect sizes in each subset is lower
than that for the data as a whole, it shows that the variable
used to split the samples is a moderator variable which
causes the differences in average effect size.
Some IS researchers have inspected potential
moderators (job involvement, self-esteem, gender) for the
relationship between role stressors and job satisfaction.
Job involvement is defined as “the psychological
identification of an individual with a specific job” [25, pp.
177]. Based on a sample of 464 IS professionals holding
various job positions, Igbaria and colleagues [24] found
that the impact of role stressors on IS employees differed
significantly among different levels of job involvement
(low, moderate, and high). Particularly, the negative
consequences of both RA and RC are stronger among the
moderate and high job involvement subgroups than for
the low-involvement subgroup. The authors concluded
that:
these findings are analogous to those documented in the
work-family literature that high levels of job involvement
and family involvement heighten the level of workfamily conflict experienced by individuals and its
aversive effects on individuals’ well-being” [24, p. 193].
Prior research has also found self-esteem to moderate
the relationship between role stressors and psychological
and physical strain [29]. One IS study based on a sample
of 124 system developers revealed that self-esteem
moderates the relationship between role stress and job
satisfaction [44]. Nevertheless, the authors didn’t mention
the direction of the moderating effect in either their
hypotheses or results.
Igbaria and Chidambaram [23] investigated the impact
of gender on various aspects of IS professionals among
348 members of the Data Processing Management
Association (DPMA) professional association. Their
results showed that women and men reported no
significant differences in role stressors. After controlling
for gender, they found that both RA and RC were still
significant predictors of job satisfaction.
Another potential moderator that may differentiate
among the observed results is the diversity in job
satisfaction measurements used for different studies.
Among the ten studies included in our meta-analysis, job
satisfaction was measured by five different scales:
0-7695-2268-8/05/$20.00 (C) 2005 IEEE
6
Proceedings of the 38th Hawaii International Conference on System Sciences - 2005
Hackman and Oldham's (1976) Job Diagnostic Survey
(JDS), Smith, Kendall and Hulin's (1969) Job Descriptive
Index (JDI), the Minnesota Satisfaction Questionnaire
Short Form (1967), measures from Hoy [16] , and revised
measures from Hoppock (1935). Some of these
instruments feature various dimensions of job satisfaction
(e.g., satisfaction with pay, promotional opportunities, coworkers, and the job itself) Yes only four studies reported
the correlation between role stressors and the specific subdimensions of their job satisfaction scales [50]. Such a
lack of specific results may inhibit the comparison
between general job satisfaction and these facets of job
satisfaction, which have been shown to differ in the
earlier meta-analysis studies [10, 28]. For example,
Jackson and Schuler’s study showed that it is necessary to
explore moderators for the relationship between RC and
general job satisfaction and also for RC and “satisfaction
with advancement opportunities,” while there was no
evidence of moderators for other sub-types of satisfaction
(satisfaction with supervision, work itself, co-workers,
and pay), because the percentage of unexplained variance
was close to zero.
Despite our identification of several potential
moderator variables, due to the minimal sample size
required in each sub-sample for conducting a moderator
analysis, and to the lack of necessary information for each
potential moderator, we were unable to conduct further
moderator analyses with the available data.
6. Conclusion
6.1. Summary of Research Findings
This paper presents the use of meta-analysis as a
statistical technique to integrate empirical IS research
results regarding the relationship between RA/RC and IS
professionals’ job satisfaction across independent studies.
In general, the results of this study found that both of the
role stressor variables – RA and RC – are negatively
correlated with IS professional’s job satisfaction across
diverse studies. The mean effect size for RA is more
strongly correlated with job satisfaction (-0.271) than the
analogous correlation between RC and job satisfaction (0.243), although the difference between them was not
statistically significant. Compared with non-IS related
jobs, IS professionals are not necessarily more- or lesssusceptible to the negative impacts of RA and RC on job
satisfaction. Also, the unexplained variance in the study
effect sizes indicates the existence of potential moderating
variables.
To understand the various demographic and contextual
factors that may differentiate various populations, future
studies should examine other, theoretically meaningful
moderators to explain the variations of both correlation
coefficients across samples of IS employees.
6.2. Limitations
It is important to clarify the limits or bounds of this
study. First, because of the lack of individual reliability
statistics from prior research [28], Hypothesis 4 is
statistically tested based on the original correlations,
instead of the weighted correlations after correcting for
sampling error and measurement error. Second, due to the
relatively small number of studies in the IS literature,
compared with the minimum sample size required for
moderating analysis, we were unable to conduct subgroup
moderating analysis in our study.
6.3. Implications and Contributions
This study is the first attempt in IS literature to
synthesize empirical study results for the correlations of
RA/RC with job satisfaction . The meta-analysis method
allows us to take every available empirical study into
consideration, overcoming the limitations of traditional
narrative reviews and provides a quantitative measure of
the relationship in question. Nevertheless, as advocated
by Hwang [19], the purpose of meta-analysis is by no
means to “determine the final word”, but “to take stock
and provide direction for future research” (p. 44). Despite
the limitations mentioned above, our research seeks to
provide future research directions in the following areas.
6.3.1. Research Directions for RA/RC Area. Past IS
studies have reported different levels of the relationship
between RA/RC and job satisfaction. The fact that our
meta-analysis supports both Hypotheses 1 and 2 in this
study helps to reduce this confusion. The lack of support
for Hypothesis 3 indicates that both RA and RC have
similar levels of negative impact on job satisfaction.
Thus, these two role stressors are both important
predictors of IS professional’s job satisfaction.
This meta-analysis study also contributes to IS
research by identifying the need to explore theoretically
based moderating variables to account for the different
levels of relationship between RA/RC and general job
satisfaction. The large unexplained variance in the mean
effect sizes is consistent with the two organizational
behavior studies, which also advocated the need for
identifying moderators for the relationship between
RA/RC and other dependent variables (including selfrated and superior-rated job performance, tension/anxiety,
and turnover intentions [10, 28]. Even though our study
only examined one dependent variable (job satisfaction),
considering the support for Hypotheses 4a and 4b (IS
professionals are not necessarily less susceptible to the
negative impacts of RA and RC on job satisfaction than
other occupations), it is possible that the correlations
between RA/RC with other outcome variables in IS
setting are also sensitive to certain moderators. Thus, IS
researchers should not limit their future research
0-7695-2268-8/05/$20.00 (C) 2005 IEEE
7
Proceedings of the 38th Hawaii International Conference on System Sciences - 2005
regarding RA and RC to simple, bivariate relationships,
but should examine possible moderators such as gender,
job duties, self-efficacy, tolerance of ambiguity, need for
structure and other personal attributes.
Another future research direction is the investigation
of the correlation between role stressors and the specific
sub-dimensions of the job satisfaction scales (e.g.,
satisfaction with co-workers, pay, promotional
opportunities, and the job task itself), since the
organizational behavior meta-analyses [10, 28] showed
that RA/RC differ in terms of their effect sizes with
general job satisfaction versus other facets of job
satisfaction.
6.3.2. Application of Meta-analysis in IS Field. The
fourth issue is related to the broader application of metaanalysis method throughout the IS field than with the
specific research area of RA and RC.
6.3.2.1 Misuse of the Term Meta-Analysis. It is not the
focus of this research to provide a comprehensive
methodological review of the use of meta-analysis in IS
research, which has been covered quite thoroughly by
Hwang [19]. Nonetheless, there is one problem that has
emerged recently (from the mid-to-late 1990s) in IS
research – the misuse of the term meta-analysis. We used
the terms “meta-analysis” or “meta analysis” as keywords
to search for IS-related, published meta-analysis research
in several online databases and conference proceedings
(as describe in the “Literature Search” section) and
located 21 studies. All 21 studies had “meta-analysis” or
“meta analysis” in their article titles or abstracts, however
2
nine of these studies did not actually perform a true
meta-analysis.
Before we mention these studies, a restatement of the
meta-analysis definition will help in clarifying the
criterion used to judge whether a study constitutes a true
meta-analysis. The term meta-analysis refers to “the
analysis of analyses … the statistical analysis of a large
collection of analysis results from individual studies for
the purpose of integrating the findings” [11, p. 3]. This
definition indicates that in order to be considered a metaanalysis study, the study should: 1) collect quantitative
results from multiple prior studies; 2) perform statistical
analysis on these quantitative data; and 3) focus on
analysis of effect size, an indicator of the strength of the
relationship between the relevant research variables [11].
The most common measures of effect sizes are Pearson’s
r (for survey research) and Cohen’s d (for laboratory
research).
2
Among the 21 studies, three of them are review papers for
meta-analysis itself, nine of them didn’t employ true metaanalysis procedures, the other nine of these studies (which are
denoted with ** in the reference) did actually perform a true
meta-analysis.
The three criteria that we identified are used here to
evaluate the nine papers that do not meet Glass’s [11]
definition of meta-analysis, defined above. Among the
nine papers, two of them first qualitatively reviewed
existing research to either develop a research framework
[11] or to identify information system success (ISS)
factors [36], and then collected first-hand data to either
evaluate the framework or study the rank order of the ISS
factors. Similarly, three other papers [15, 33, and 43] only
conducted a qualitative review of prior research, with no
quantitative data analysis involved. These five studies
satisfied none of the three meta-analysis criteria. In all
cases, the authors’ use of the term “meta-analysis” in their
paper titles or abstracts actually referred to the traditional
qualitative literature review – which Hunter and Schmidt
[18] label as “narrative review.”
The other four papers met the first two criteria, in that
they gathered quantitative data from prior research and
performed some statistical analysis on these data.
However, none of these studies considered effect sizes as
the focus of their study. That is, none of them actually
integrated statistical results from prior studies’ research
variables. Instead, they examined other characteristics of
the secondary data. Palvia and his colleagues [45]
collected secondary data about the ranking of key IT
issues, and then used cluster analysis to regroup them and
used t-tests to verify the categories. Chau [6] investigated
the relationship between construct reliability and research
design characteristics using ANOVA to identify which
design characteristics affect reliability. Esteves and
Ramos [9] provide a descriptive statistical overview of IS
studies conducted in Portugal. Kohli and Devaraj [34]
employed logistic regression and discriminant analysis to
examine the structural variables that affect IT payoff.
While the last study seems closest to a true meta-analysis,
the authors did not re-analyze the original effect sizes
(correlation coefficients) from the original studies; rather,
they simply counted the number of prior studies with
positive and significant, zero, or non-significant findings
as their form of analysis and conducted some analysis on
the numbers of studies in each category. They did not
examine the actual strength of the correlation, regression,
or Lisrel paths in the prior studies and thus, their study
neglected to analyze true effect sizes.
These examples of studies that mis-use the term metaanalysis show that this method is not yet well understood
by the IS research community. This is unfortunate, since
there are meta-analyses in the IS literature going back
over a dozen years [1]. An unexpected but noteworthy
contribution of our study is to raise this issue and
emphasize the correct use of the term, meta-analysis,
which is elaborated in the following section.
6.3.2.2. Correct Use of the Term Meta-Analysis. The
evaluation of the papers that do not meet Glass’s [11]
definition of meta-analysis indicates that the meaning of
0-7695-2268-8/05/$20.00 (C) 2005 IEEE
8
Proceedings of the 38th Hawaii International Conference on System Sciences - 2005
meta-analysis has been diluted, so that some researchers
tend to mis-label two other types of studies as metaanalysis. The first type is the traditional qualitative
literature review or narrative review (e.g., [11], [36], [15],
[33], and [43]). This is by no means intended to de-value
the contributions that narrative reviews can make, but
there are certain circumstances under which a qualitative
review is unable to provide an accurate representation of
population characteristics. One such situation is when the
number of prior studies is large and the studies are not
comparable in terms of sampling, measures, and so forth.
This limitation can, to certain degree, be well handled by
meta-analysis. For example, Hunter and Schmidt
recommended procedures that can “deal with variations in
study effect size due to sampling error and other artifacts
and with attenuation of effect-size estimates due to
measurement unreliability and range restriction” [18, p.
485]. Thus, qualitative literature review and meta-analysis
are two different methods which are suited for different
objectives – the term meta-analysis is not a synonym for
qualitative literature reviews.
The second type of study that often being mis-labeled as
meta-analysis is research that does include quantitative
analyses based on existing literature, however, the
quantitative analysis does not focused on prior studies’
research variables (e.g., [45], [6], [9], and [34]). Again,
we have no intent to discredit the value of this type of
study. Nevertheless, the emphasis of these studies is
substantially different from the objective of metaanalysis. Therefore, the term meta-analysis is not
interchangeable with this type of research.
In conclusion, we advocate the term meta-analysis should
only be used for those studies that: 1) collect quantitative
results from multiple prior studies; 2) perform statistical
analysis on these quantitative data; and 3) focus on
analysis of effect size, an indicator of the strength of the
relationship between the actual research variables from
those prior studies[11].
7. References3
[1] Alavi, M. and Joachimsthaler, E. A. “Revisiting DSS
Implementation Research: A Meta-Analysis of the
Literature and Suggestions for Researchers,” MIS
Quarterly, 16:1, March 1992, pp. 95-116.
[2]*Baroudi, J.J. “The Impact of Role Variables on IS
personnel Work Attitudes and Intentions,” MIS Quarterly,
December 1985, pp. 341-356.
[3] Bartol, K.M., and Martin, D. "Managing Information
Systems Personnel: A Review of the Literature and
3
The papers denoted with * were included in our quantitative
meta-analysis. The papers denoted with ** were the prior
studies in the IS literature that did employ true meta-analysis
procedures.
Managerial Implications," MIS Quarterly (6), December
1982, pp. 49-70.
[4]*Bostrom, R. P. "Role Conflict and Ambiguity:
Critical Variables in the User-Designer Relationship,"
Proceedings of the Seventeenth Annual Computer
Personnel Research Conference, Association for Computing
Machinery Press, New York, 1981, pp. 88-112.
[5]**Cao, Q., Nah, F.F., and Siau, K. “A Meta-Analysis
on Relationship Modeling Accuracy: Comparing
Relational and Semantic Models”, Proceedings of AMCIS
Conference, 2000, Long Beach, CA, 470-473.
[6] Chau, P.Y.K. “On the Use of Construct Reliability in
MIS Research: A Meta-Analysis,” Information &
Management (35), 1999, pp. 217-227.
[7] Cohen, T.D. Statistical Power Analysis for the
Behavioral Sciences, (2nd ed.). Jillsdale, NJ: Lawrence
Erlbaum Associates, 1988.
[8]**Dennis, A.R., Haley, B.J., and Vandenberg, R.J. “A
Meta-Analysis of Effectiveness, Efficiency, and
Participant Satisfaction in Group Support Systems
Research”, Proceedings of ICIS Conference, 1996,
Cleveland, OH, 278-289.
[9] Esteves, J. and Ramos, I. “A Meta-Analysis of
Information Systems Research Diversity in Portugal”,
Proceedings of AMCIS Conference, 2003, Tampa, FL,
966-974.
[10] Fisher, C.D. and Gitelson, R. “A Meta-Analysis of
the Correlates of Role Conflict and Ambiguity,” Journal
of Applied Psychology (68:2), 1983, pp. 320-333.
[11] Glass, G.V. “Primary, secondary, and metaanalysis,” Educational Researcher (5), 1976, pp. 3-8.
[12]*Goldstein, D. K., and Rockart, J. F. "An
Examination of Work-Related Correlates of Job
Satisfaction in Programmer/Analysts," MIS Quarterly (8),
1984, pp. 103-115.
[13] Greenhaus, J.H. and Beutell, N.J. “Sources of
Conflict Between Work and Family Roles,” Academy of
Management Review (10:1), 1985, pp. 76-88.
[14]*Guimaraes, T. and Igbaria, M. “Determinants of
Turnover Intentions: Comparing IC and IS personnel,”
Information Systems Research (3:3), 1992, pp. 273-303.
[15] Heine, M.L., Grover, V. and Malhotra, M. K. “The
Relationship between Technology and Performance: A
Meta-Analysis of Technology Models,” Omega (31:3),
June 2003, pp. 189-204.
[16] Hoy, F. Small Business Survey Questionnaire.
Athens, GA: Small business Development Center
Publications, The University of Georgia, 1983.
[17] Huarng, A. S. “Burnout syndrome among
information system professionals,” Information Systems
Management (18:2), Spring 2001, pp. 15-xx.
[18] Hunter, J.E. and Schmidt, F.L. Methods of MetaAnalysis, Correcting Error and Bias in Research
Findings. Thousand Oaks: Sage Publication, 1990.
0-7695-2268-8/05/$20.00 (C) 2005 IEEE
9
Proceedings of the 38th Hawaii International Conference on System Sciences - 2005
[19] Hwang, M.I. “The Use of Meta-Analysis in MIS
Research: Promises and Problems,” The DATA BASE for
Advances in Information Systems (27:3), 1996, pp. 35-48.
[20]**Hwang, M. “Did Task Type Matter in the Use of
Decision Room GSS? A Critical Review and a MetaAnalysis,” Omega (26: 1), 1998, pp. 1-15.
[21]**Hwang, M.I. and Wu, B.J.P. “The Effectiveness of
Computer Graphics for Decision Support: A MetaAnalysis Integration of Research Findings,” The DATA
BASE for Advances in Information Systems (21:3), 1990,
pp. 11-20.
[22]**Hwang, M.I., Windsor, J.C., and Pryor, A.
“Building a Knowledge Base for MIS Research: A MetaAnalysis of a Systems Success Model,” Information
resources Management Journal (13:2), 2000, pp. 26-32.
[23]*Igbaria, M. and Chidambaram, L. “The Impact of
Gender on career success of information systems
professionals:
A
Human-Captial
Perspective,”
Information Technology & People (10:1), 1997, pp. 6386.
[24] Igbaria, M., Meredith, G., and Smith, D.C.,
“Predictors of Intention of IS Professionals to Stay With
the Organization in South African,” Information &
Management (26), 1994, 245-256.
[25]*Igbaria, M., Parasuraman, S. and Badawy, M.K.
“Work Experiences, Job Involvement, and Quality of
Work Life Among Information Systems Personnel,” MIS
Quarterly, June 1994, 175-201.
[26] Ivancevich, M., Napier, A. H., and Wetherbe, J. C.
"Occupational Stress, Attitudes, and Health Problems in
the Information Systems Professional," Communications
of the ACM (26:10), 1983, pp. 800-806.
[27]*Ivancevich, M., Napier, A. H., and Wetherbe, J. C.
“An Empirical Study of Occupatoinal Stress, Attitudes
and Health Among Information Systems Personnel,”
Information & Management (9), 1985, pp. 77-85.
[28] Jackson, S.E. and Schuler, R.S. “A Meta-Analysis
and Conceptual Critique of Research on Role Ambiguity
and Role Conflict in Work Settings,” Organizational
Behavior and Human Decision Processes (36), 1985, pp.
16-78.
[29] Jex, S. and Elacqua, T. “Self-esteem as a Moderator:
A Comparison of Global and Organization-based
Measures,” Journal of Occupational & Organizational
Psychology (72:1), 1999, pp. 71-81.
of Organizational & End User Computing (16:1), 2004,
pp. 59-72.
[42]**Mahmood, M.A., Hall, L., and Swanberg, D.L.
“Factors Affecting Information Technology Usage: A
Meta-Analysis of the Empirical Literature,” Journal of
Organizational Computing and Electronic Commerce
(11:2), 2001, pp. 107-130.
[43] Marshall, C., Shipman, F. M., and McCall, R. J.
“Making Large-Scale Information Resources Serve
[30] Kahn, R.L., Wolfe, D., Quinn, R., Snoek, J.D., and
Rosenthal, R. Organizational Stress: Studies in Role
Conflict and Role Ambiguity, John Wiley and Sons, New
York, New York, 1964.
[31] King, R.C. and Sethi, V., “The Impact of
Socialization on the Role Adjustment of Information
Systems Professionals,” Journal of Management
Information Systems (14:4), Spring 1998, pp. 195-217.
[32]*King, R.C. and Xia, W.D., “Retaining IS Talents in
the New Millennium: Effects of Socialization on IS
Professionals’ Role Adjustment and Organizational
Attachment,” Proceedings of the ACM SIGCPR, San
Diego, CA., 2001, 144-156.
[33] Klaus, H., Rosemann, M. and Gable, G.G. “What is
ERP?” Information Systems Frontiers (2:2), 2000, pp.
141-162.
[34] Kohli, R. and Devaraj, S., “Measuring Information
Technology Payoff: A Meta-Analysis of Structural
Variables in Firm-Level
Empirical Research,”
Information Systems Research (14:2), 2003, pp. 127-145.
[35]*Lee, P.C.B., “Turnover of Information Technology
Professionals: a contextual model,” Accounting,
Management and Information Technology (10),, 2000,
101-124.
[36] Li, E.Y. “Perceived Importance of Information
System Success Factors: A Meta Analysis of Group
Differences,” Information & Management (32), 1997, pp.
15-28.
[37]*Li, E.Y., and Shani, A. B. "Stress Dynamics of
Information Systems Managers: A Contingency Model,"
Journal of Management Information Systems (7:4), 1991,
pp. 107-130.
[38]**Lim, J., Richardson, V.J., and Roberts, T.L.
“Information Technology Investment and Firm
Performance: A Meta-Analysis,” Proceedings of the 37th
HICSS, 2004, HI, pp. 1-10.
[39] Lipsey, M.W., & Wilson, D.B. (2001). Practical
Meta-analysis. Thousand Oaks: Sage.
[40] Locke, E. “The Nature and causes of Job
Satisfaction,” in Handbook of Industrial and
Organizational Psychology, M.D. Dunnette (ed.), Rand,
Chicago, Illinois, 1976, pp. 1297-1349.
[41]**Ma, Q. and Liu L. “The Technology Acceptance
Model: A Meta-Analysis of Empirical Findings,” Journal
Communities of Practice,” Journal of Management
Information Systems (11: 4), Spring 1995, pp. 65-86.
[44] Nelson, A.C. and LeRouge, C., “Self-Esteem:
Moderator Between Role Stress Fit and Satisfaction and
Commitment?,” Proceedings of the ACM SIGCPR,
2001, San Diego, CA, pp. 74-77.
[45] Palvia, P.C., Palvia, S.C.J., and Whitworth, J.E.,
“Global Information Technology: A Meta-Analysis of
Key Issues,” Information & Management (39), 2002, pp.
403-414.
0-7695-2268-8/05/$20.00 (C) 2005 IEEE
10
Proceedings of the 38th Hawaii International Conference on System Sciences - 2005
[46] Premkumar, G., “A Meta-Analysis of Research on
Information Technology Implementation in Small
Business,” Journal of Organizational Computing &
Electronic Commerce (13:2), 2003, pp. 91-121.
[47] Rizzo, J.R., House, R.J., and Lirtzman, S.I., “Role
Conflict and Ambiguity in Complex Organizations,”
Administrative Science Quarterly (15:2), 1970, pp. 150163.
[48] Rosenthal, R. “The “file drawer problem” and
tolerance for null results,” Psychological Bulletin (86),
1979, pp. 638-641.
[49]**Sharma, R. and Yetton, P. “A Contingent Model
for the Role of Management Support in IS
Implementation: A Meta-Analysis”, Proceedings of
ICIS Conference, 1997, Atlanta, GA, 492-493.
[50] Smith, P., Kendall, L. and Huliln, C., The
Measurement of Satisfaction in Work and Retirement,
Rand, Chicago, Illinois, 1969.
[51] Viswesvaran, C., and Ones D.S., “Theory Testing:
Combining Psychometric Meta-Analysis and Structural
Equations Modeling,” Personnel Psychology (48), 1995,
865-885.
0-7695-2268-8/05/$20.00 (C) 2005 IEEE
11