Experience and Scale and Scope Economies: Tradeoffs and
Performance in Drug Development
CHRISTOPHER S. BOERNER
1 DNA Way
Genentech Incorporated
South San Francisco, CA 94080
650-225-1369 (O)
650-225-1077 (F)
[email protected]
and
JEFFREY T. MACHER*
G-04 Old North
McDonough School of Business
Georgetown University
Washington, DC 20057
202-687-4793 (O)
202-687-4236 (F)
[email protected]
Abstract: This paper examines whether firm experience and scale and scope economies in
technological and clinical areas of drug development influence the speed with which Contract
Research Organizations (CROs) complete new drug development projects for pharmaceutical
firms. The empirical results suggest that greater experience in specific technological areas and
scope economies in related technological areas have significant effects on performance, although
there is variation across development phases. The results provide a more nuanced examination of
the drivers of performance improvement in firms’ innovative activities, as well as suggest that
both experience and scope economies are important factors.
Keywords: firm experience, scale economies, scope economies, technological innovation
*
This research was supported in part from funding from The Alfred P. Sloan Foundation. We wish to acknowledge the
individual comments of David Mowery, Nick Argyres, Iain Cockburn, Jeff Furman and Anita McGahan, as well as four
anonymous referees from the 2002 Academy of Management Conference in Denver, CO. The authors are solely responsible for
any errors or omissions. Address all correspondence to Jeffrey Macher at the above address.
1.
INTRODUCTION
Over the last two decades, a growing body of research has emerged that documents and
attempts to explain research and development (R&D) related performance differences across
firms in a variety of industries. One research approach argues that through a cumulative activity
of exploitation (Levinthal and March 1993; March 1991), firms create competitive advantage
from experiential learning (Baum, Li et al. 2000) and localized firm knowledge and expertise
(Leonard-Barton 1992; Wernerfelt 1984). Superior innovative performance results from the
informational advantages and reductions in uncertainty that result from greater firm experience.
In contrast, research in the economics of R&D and innovation (Cohen 1995; Griliches
1995; Patel and Pavitt 1995) suggests that innovative performance is conditioned in large part by
the size and breadth of firms’ innovative activities, such as scale and scope economies. These
explanations for differential R&D performance have a long and prestigious history, arguably
tracing their roots back to the Schumpeterian hypothesis of increasing returns in R&D
(Schumpeter 1951).1 And this approach has spawned a large body of literature (Galbraith 1952;
Henderson and Cockburn 1996; Nelson, Peck et al. 1967; Pavitt 1987; Scherer 1965) that
explores whether there are advantages to firm size and scope in the conduct of R&D activity
itself. Despite this research, theoretical and empirical ambiguity remains over how precisely the
scale and scope of firms affect innovative performance.
In this paper, we compare and contrast these two perspectives in an examination of the
importance of experience and scale and scope on firms’ innovative performance. We utilize a
unique data set containing project-level data from pharmaceutical Contract Research
Organizations (CROs) to explore the role of these factors in determining the speed with which
new products are able to move through the drug development process. Over the last two decades,
CROs have become an increasingly important part of drug development and the evolving
industrial organization of the pharmaceuticals industry. These organizations are now involved in
virtually all aspects of drug development, and account for roughly one-third of total industry
R&D spending (CMR_International 1999). Employing project- and firm-level data, we analyze
how firm experience and scale and scope economies influence development performance. We
measure performance as the time with which drug development projects are completed by CROs,
1
See Stoneman (1995) for a comprehensive review of the literature in this area.
1
which is an increasingly important performance dimension and considered an accurate proxy for
the costs associated with drug development.2 The availability of detailed project-level data by
clinical phase allows for not only the development of accurate proxies for these factors, but also
an examination of which factors are more important depending upon the development stage.
Within this setting, we therefore are able to more directly assess the effects of these factors—as
well as examine if any tradeoffs exist between and among these factors—on innovative
performance.
This paper seeks to contribute to research that examines firms’ innovative activities
within the economics of R&D. A more detailed examination of the importance of firm
experience and scale and scope economies is possible because of the detailed “technological
area” data that exist. Because we employ data from various drug development stages, we are also
able to identify factors that are more important to performance early in a new product
development cycle versus those that are relevant closer to when products reach the market. Such
microeconometric evidence of the effects of such factors is quite rare in the economics of R&D
literature (Henderson and Cockburn 1996). Finally, the paper also examines an area of
innovative activity—pharmaceutical drug development—that has received more limited attention
(Azoulay 2002; Bogner and Thomas 1994; Cockburn and Henderson 2001), especially in
comparison to research that examine pharmaceutical research (Cockburn and Henderson 1994;
Cockburn, Henderson et al. 1999; 2000; Henderson 1994; Henderson and Cockburn 1994; 1996).
We note that the shift in the nature of drug discovery from “random” to “rational” has
helped to separate the requirements underlying the drug discovery process from those required
for drug development. With this de-coupling has come a dramatic increase in the prevalence of
CROs concentrated in a broad range of drug development activities. By highlighting the
relationships between the scale and scope of CROs’ activities and technological area experience,
this paper provides important insights into the growth of CROs, as well as their place in the
evolving industrial organization of the pharmaceuticals industry.
2
First mover advantages and intense post-patent generic competition give pharmaceutical companies strong incentives to get
new drug products to market quickly. Case study evidence suggests that when virtually identical products are brought to market
as little as three to six months apart, the product appearing first acquires the largest share of the market and maintains this
advantage indefinitely (Wiggins 1981). Even when a new drug does not face initial competition from other products, time to
market is still strategically important since the majority of the returns associated with a new product are reaped during the
window of time between regulatory approval and patent expiration (Grabowski and Vernon 1990; OTA 1993).
2
The next section of the paper examines the pharmaceutical industry, highlighting in
particular the distinction between drug discovery and drug development, the pharmaceutical
industry’s evolving industrial organization, and the increasingly important role of CROs. Section
3 reviews the strategy and economics literature related to R&D and innovation. This section then
frames the importance of firm experience and scale and scope economies for pharmaceutical
drug development, articulating the hypotheses and empirical implications that result. The
empirical approach is presented in Section 4, while Section 5 presents the econometric results,
including the economic significance and limitations of the analysis. Section 6 makes concluding
comments.
2. BACKGROUND
2.1. Separation of Drug Discovery and Development
There are two main activities required to bring a new drug product to market in the
pharmaceuticals industry. The first set of activities—commonly referred to as drug discovery—
involves the initial screening and extraction of a molecule or compound with certain desired
therapeutic properties. Once a new substance has been “discovered,” a second set of
development-oriented activities that involve extensive laboratory, animal and human testing and
considerable regulatory oversight begins. Although the main phases of drug development are
applied to virtually every new drug product in the United States,3 the specific activities
undertaken in each of these phases vary considerably by product.4 The most important issues to
3
Drug development involves a number of non-clinical and clinical activities. Non-clinical activities include the development
of basic drug profiles, formulation of appropriate dosage levels, and demonstrating safety to begin testing in humans.
Pharmacologic tests to define the drugs basic mechanism of action and dose response, and toxicological tests to determine
harmful effects of various dosage levels are used. Clinical activities are separated into phases. Phase 1 activities include the
cautious administration of the drug to between 20 and 80 healthy patients, in order to determine the metabolism and
pharmacologic action of the drug in humans so as to further demonstrate and assess the drug's clinical safety. Phase 2 activities
include the well-controlled and closely monitored administration of the drug to typically a few hundred patients suffering from
disease or ailment that the product is meant to address. Important tests include determining the appropriate dose and regimen, as
well as evaluating the effectiveness with respect to key endpoints. Phase 3 activities include evaluation of the overall benefit /
risk relationship of the drug, on anywhere from several hundred to several thousand patients in many different testing centers in
multiple countries. Important tests include identifying potential adverse drug reactions (both short-term and long-term),
establishing the drug's efficacy, and determining appropriate dosage levels. Throughout the three phases, frequent interaction
with regulators at FDA is required. Post-Approval Research in the form of experimental studies and surveillance activities are
normally undertaken after a drug has been approved.
4
FDA regulations covering new drug development leave most of the key decisions regarding how to proceed in the hands of
the sponsoring firm (O.T.A. 1984). Rather than providing a recipe for new product development, regulations specify a series of
hurdles that companies must clear in order to gain FDA approval. In reviewing a New Drug Application (NDA), FDA focuses on
(a) whether the drug is safe in its proposed use; and (b) whether the firm (or its contractors) can effectively manufacture the
product. If FDA approves the NDA, the firm is free to begin marketing the drug product. In addition, sponsoring firms must
3
Food and Drug Administration (FDA) regulators are (1) ensuring that sponsor firms have
developed accurate profiles of drugs (i.e., basic activity, dose response, mechanism of action,
etc.); (2) determining that sponsor firms have proven drug safety for human consumption and
effectiveness in their proposed uses; and (3) verifying that sponsor firms—including contract
firms—can manufacture the products produced while preserving the drug composition and
stability.5
Throughout the pharmaceutical industry, close linkages have existed between drug
discovery and development, as scientists lacked a well-developed corpus of knowledge regarding
the biological underpinnings of specific diseases and were forced to rely more on observation
than on established theory (Spilker 1989). Drug discovery was effectively a quasi-random
screening process in which a large number of natural and chemically derived compounds were
randomly tested in laboratory and animal models for potential therapeutic activity. To validate
working hypotheses based on prevailing knowledge and historical observation, promising
compounds were synthesized and subjected to extensive in vitro and animal testing. Assuming
the compounds had the desired effect in these relatively simple contexts, researchers would then
commence clinical testing to explore safety and efficacy in the far more complicated human
setting.6 The close linkage between discovery and development existed as both activities relied
on similar sets of activities, including therapeutic experience in a given drug class, the ability to
rapidly screen a large number of drug compounds against a given target, the possession of a
network of pre-clinical and clinical testing resources, and the ability to assimilate and analyze the
resulting testing data. In this environment, it was difficult to disentangle firm performance in
drug development from performance in drug discovery.
Over the last 25 years, however, two major advances have altered the status quo and led
to significant changes in the R&D process (Malerba and Orsenigo 2001), and subsequently,
industry structure. After decades of publicly funded biomedical research, substantial scientific
advances in the fields of physiology, pharmacology, enzymology, and cell biology were made.
abide by a host of regulations affecting laboratory, clinical and manufacturing practices. These regulations outline the sponsoring
firm's (as well as its contractors') specific responsibilities with respect to laboratory and clinical tests and manufacturing.
5
Additional regulations set basic operating standards for the laboratories, personnel and manufacturing facilities involved in
the various phases of drug development. These regulations take the form of Good Manufacturing Practice, Good Clinical Practice
and Good Laboratory Practice regulations. For a more lengthy discussion of these issues, see Mathieu (1997).
6
Throughout this process, results from a given series of tests would frequently lead to modifications or outright rejection to
the underlying hypotheses. It was not uncommon, for instance, for researchers to find through clinical testing that a compound
designed to treat one indication was far more successful at treating another.
4
These advances not only provided better understanding of the “mechanisms of action” of many
existing drugs, but also facilitated the drug discovery process itself by allowing researchers to
more carefully screen, select, and test a wider range of drug compounds. These combined effects
gave rise to what is commonly known as “rational drug design,” or the application of biomedical
knowledge to the design of new compounds, as well as to the ways in which these compounds
are tested (Cockburn and Henderson 1999; Malerba and Orsenigo 2001). The second advance
that has altered the conjoined approach to drug discovery and development was the rise of
biotechnology and the new opportunities it has created for pharmaceutical research.7 Although
firms have utilized genetic engineering to manufacture previously understood proteins in large
enough quantities to permit their use as therapeutic agents, they are now increasingly relying on
advances in biotechnology to enhance the discovery of entirely new synthetic chemical drugs
(Malerba and Orsenigo 2001).
Both of these advances have reduced the degree of uncertainty involved with drug
research and have allowed scientists to better design target compounds through more established
scientific theory (Arora and Gambardella 1994), thereby decreasing the centrality of subsequent
in vitro and in vivo testing to the discovery process.8 Advances in biomedical knowledge have
helped decouple the competencies that underlie drug discovery from those required for drug
development. Since the knowledge and expertise that are critical to drug discovery can be more
easily codified and partitioned, it is no longer the case that discovery and development activities
are more effectively performed together. Opportunities for smaller, less established firms to enter
into discovery and development activities have subsequently increased over time (Arora and
Gambardella 1994), leading to increased vertical specialization in pharmaceutical innovation.
2.2. Rise of Contract Research Organizations (CROs)
The effective separation of the skills required for drug discovery and development have
led to an increase in the number of small, specialized drug development firms. Since the early
1980s, an estimated one-third of industry R&D spending on clinical trials is now conducted by
independent CROs (CMR_International 1999). The separation of requirements described above
7
These opportunities started with the discovery of the double-helix structure of DNA in the mid-1950s and continued with
subsequent discoveries during the 1970s of techniques of recombinant-DNA and cell fusion, among others.
8
Gambardella (1995), for instance, highlights a number of high-profile drug products, including Mevacor, Tagamet and
Prozac, which were discovered only after their basic molecular structure had been outlined utilizing the sort of biomedical
knowledge outlined above.
5
therefore allows for an examination of the firm-level factors that drive performance in new drug
product development independent from those involved in drug discovery.
Several aspects of drug development make it an ideal setting in which to examine the
effects of experience and scale and scope economies on firm performance. Throughout the
course of drug development, massive amounts of information and data are generated, including
the results of scores of laboratory and animal tests, as well as detailed information gathered from
thousands of patients. As the requirements for clinical trials have increased in recent decades,9 so
too has the amount of data that must be collected, checked, analyzed, and archived. At each stage
in the drug development process, firms require systems in place to effectively gather and analyze
the large amounts of data produced. Moreover, stringent FDA auditing processes are required to
ensure data accuracy. As Spilker (1989) notes, drug development is fundamentally a planning
exercise whereby information obtained in one stage must be reviewed and analyzed to determine
how best to proceed in terms of allocating resources in subsequent stages over multiple years
involving multiple organizations. Simply managing the information associated with these
networks of patients and clinicians can entail significant investment on the part of CROs.
Moreover, pharmaceutical and contract firms are increasingly attempting to track the clinical
outcomes of patients after a new drug has been marketed. This information has the potential to
provide significant value to firms in developing follow-on drugs that are more effective or entail
fewer side effects. Firms may also use such information to gain insights into potential new uses
for a marketed product. Yet collecting post-approval clinical outcome data involves managing
information from a considerably larger network of providers and patients.10 Finally, contract
research firms must manage the collection and exchange of information across multiple
locations, as clinical trials are increasingly being conducted at multiple hospitals and testing
facilities spread across many different geographic regions. The information obtained from these
various sources must be made available in various formats to personnel at both the CRO and
sponsor firm, who are also spread across multiple and geographically-isolated locations. The
need to collect, make accessible, or exchange information across many different locations—in
9
Since the early 1970s, the number of clinical trials per new drug application (NDA) has increased from 30 trials to over 70
trials today. These trials involve increasingly larger numbers of patients, increasing from 500 patients in 1970 to roughly 4,000
today (PhRMA 1999). On average, each of these patients undergoes nearly 150 clinical tests, with the results of these tests
eventually submitted to regulators as part of the formal request for marketing approval for a new drug application (NDA). The
average NDA consists of roughly 30 volumes and more than 100,000 pages of data.
10
In addition to the technical challenge there are also significant legal and regulatory issues as well.
6
real-time and over diverse technology platforms—is significantly complex and requires
extensive networking and data management capabilities, at least providing some rationale for the
importance of size in this process.
Clinical development experience is another factor critical to drug development
performance, as development approaches vary significantly according to the therapeutic
characteristics of the drug product. The difficulties associated with the development of new
protocols for drugs in the early stages of development suggest that localized experience and
established knowledge bases within specific technological areas or in specific phases of
development may offer performance benefits. Similar arguments can be made for the importance
of scope economies in technological areas. We elaborate on both factors below.
3. HYPOTHESES
This paper examines product development in a single industry, pharmaceuticals, and by a
single organizational form, contract research organizations (CROs). As described above, the
existence of specialized CROs provides an ideal setting to isolate the factors important for drug
development independent of those for drug discovery, as CROs focus almost entirely on drug
development activities. Within this setting, we further limit our examination to a set of clinicalrelated activities—technological and clinical area experience and scale and scope economies—
that we are able to generate good proxies for. Although a narrow focus perhaps limits the
generalizability of our results, it allows for greater precision in our measures and a more direct
link between the presence of these factors and differences in firm performance.
3.1. Firm Experience
Organizations are often described as routine-based and history-dependent systems that
adapt incrementally to past experiences (March and Simon 1958). Deeply embedded
organizational routines both enable and constrain what firms can do (Baum, Li et al. 2000).
While prior research has examined differences between firms exploring new routines and
exploiting existing routines (Levinthal and March 1993; March 1991), competence building is
usually a cumulative activity of exploitation, whereby experiential learning improves from
greater focus in particular areas (Baum, Li et al. 2000). Because superior competence typically
arise through experiential learning that is based on embedded or highly tacit knowledge, it is not
7
only largely immobile and difficult for other firms to easily acquire or imitate (Teece 1982), but
also likely to be highly specific to functional, technological or product areas. These
characteristics suggest that localized firm knowledge and expertise can be important sources of
sustainable competitive advantage (Leonard-Barton 1992; Wernerfelt 1984). In particular,
experiential learning likely provides informational advantages or gains via the perceived
legitimacy (March 1988) or past success (Baum, Li et al. 2000) of repetition, or the
accompanying reduction in uncertainty from learning by doing (Argote 1999). By contrast, firms
relatively new to specific areas are unlikely to have developed heuristics in place for correctly
making investments or strategic decisions, or for effectively analyzing the relevant information
that exists (Henisz and Delios 2001). Research that examines firm experience has been largely
supportive of these claims, including studies in populations of commercial banks (Pennings and
Harianto 1992), investment banks (Podolny 1994) and acquiring firms (Haleblian and Finklestein
1999); technological choices among firms (Christensen and Bower 1996; Stuart and Podolny
1996); and foreign market entry (Delios and Henisz 2000; Mitchell, Shaver et al. 1992), among
others.
While there are multiple areas in which firms may develop important and strategically
relevant experience in drug development, a review of the pharmacoeconomics literature and
discussions with practitioners highlight the importance of therapeutic area and clinical
development experience, for at least two reasons. First, because clinical development approaches
vary significantly by the therapeutic characteristics of the drug, prior experience in developing
new drug types is particularly important in determining subsequent performance in developing
drugs in the same product (or therapeutic) family. This expertise is particularly useful when
firms are developing the protocols for a new drug product that is in the early stages of the
development process. As firms become more experienced in a given technological area, they
acquire localized expertise and problem-solving strategies that can be drawn on in subsequent
and similar efforts. The existence of an established knowledge base in a given technological area,
as well as established routines for handling similar types of problems (Nelson and Winter 1982),
partially explains why some firms have acquired strong reputations for developing drugs in
particular areas.11
11
Bristol Myers, for instance, is widely recognized for its competencies in development oncology drugs (Stern 1996).
8
A second aspect of clinical development expertise relates to the more general skills and
knowledge that firms acquire as a result of prior drug development experiences. As firms
become more experienced in development, they acquire stocks of generalized technological
expertise related to the early-stage testing of new drug products, the profiling of new drugs, and
the efficient management of clinical trials. These firms also become more familiar with the
general regulatory and administrative issues associated with clinical development and the FDA.
Given that clinical trial start-up costs, regulatory demands and project management concerns are
all cited as among the major sources of delay in drug development, greater experience in these
arenas are likely differentiators of firm performance.12
As firms become more experienced in planning and conducting clinical trials, they may
acquire a storehouse of embedded expertise that can be drawn upon for subsequent drug
development projects. To the extent that this expertise builds upon knowledge that has been
acquired over time and becomes organizationally embedded, localized expertise may lead to
performance advantages and come to represent a source of lasting competitive advantage as
competitors cannot easily acquire, or even imitate, these skills.13 Greater experience is likely
idiosyncratic to not only particular therapeutic areas, but also drug indications within a given
therapeutic area. Greater therapeutic area experience, or drug indication experience within a
given therapeutic area, should therefore lead to superior performance for the next drug
development project within the therapeutic area or drug indication, respectively and ceteris
paribus. Similar arguments can be made for greater clinical phase experience.
Early phases of development, or more knowledge-intensive activities (Azoulay 2002),
require a more thorough understanding of both the nature of the target disease and the
mechanism of action of the drug designed to treat that disease in order to effectively design
clinical protocols. As drug development projects progress through the stages of clinical
development and get closer to market, therapeutic expertise arguably becomes less important
than the operational activities (e.g. data and project management) noted above. An interesting
phenomenological question therefore is to better understand where, at what level, and what type
of experience is critical in differentiating performance in drug development. We therefore
12
Practitioners at both established pharmaceutical firms and contract organizations listed these factors along with poor
protocol design as the most obvious sources of delay in executing clinical trials.
13
Merck, for example, is generally regarded as having very strong expertise in clinical trial design and management. See for
example, "Battling blockbusters," Business Week, January 11, 1999, 105.
9
examine the following general hypothesis, but explore the effects of knowledge-intensive versus
data-intensive experience and development phase experience in the econometric analysis.
H1:
Greater technological area and/or clinical phase experience improve drug
development, ceteris paribus.
3.2. Scale and Scope Economies
Frequently attributed to Schumpeter (1942), the association of firm size with scale and
scope economies, market power, and the ability to aggregate inputs is widely asserted to confer
performance advantages on large firms (Scherer 1980). This discussion has been interpreted by
many as a claim that there are increasing returns in research and development (R&D), both to
R&D establishment size and to firm size. For several reasons, size may also provide advantages
in the conduct of firms’ R&D efforts (Cohen 1995) or innovative activities (Galbraith 1952).
First, capital market imperfections confer advantages to large firms in securing financing for
R&D projects and in providing mitigation strategies against adverse selection and moral hazard
in the raising of capital. Second, large firms may be better able to spread the fixed costs of
research over a larger sales base in the absence of fully functioning markets for innovation
(Cohen and Klepper 1996). Third, large firms are able to exploit economies of scale in the
conduct of the R&D activity itself (Panzar and Willig 1981). Finally, large firms may have
greater access to the complementary technologies and downstream capabilities (i.e., marketing
and finance) that are presumed to make R&D more productive (Cohen 1995). Despite these
persuasive arguments, empirical findings are mixed (Cohen and Levin 1989; Patel and Pavitt
1995). Some researchers note that inconsistent findings result from difficulty in developing good
measures of innovation (Cohen 1995), while other researchers argue that a lack of sufficiently
detailed data make it difficult to distinguish between measures of economies of scale and
economies of scope (Henderson and Cockburn 1996).
Theoretical controversy surrounding the influence of scale economies on innovative
performance arises because counter-arguments have been made for each of these propositions
above (Scherer and Ross 1990). One central problem is that organizational costs associated with
performing R&D in larger organizations are often ignored. The loss of managerial control and
additional bureaucracy that is often associated with large organizations, for example, may
adversely affect innovative performance. Innovative activities performed in large organizations
10
may likewise dampen the incentives of individual scientists and entrepreneurs by preventing
them from capturing the benefits of their efforts (Sah and Stiglitz 1988).
Direct empirical tests of whether size confers any advantages in innovative performance
have proven challenging due to the difficulties associated with measuring scale economies in
R&D. Many scholars have instead tested whether the inputs to research and development (e.g.,
R&D expenditures) increase more than proportionately with firm size (Cohen 1995; Fisher and
Temin 1973). These efforts report that the likelihood of a firm reporting positive R&D effort
increases with firm size and approaches one for the largest firms (Bound, Griliches et al. 1984),
within-industry R&D increases monotonically with firm size, and firm size explains more than
half of the variation in R&D among R&D-performing firms (Comanor 1967; Scherer 1965).
Fisher and Temin (1973) importantly point out that examining the relationship between R&D (an
innovative input) and firm size speaks only to the amount of R&D that firms perform, but says
little about the efficiency of innovative performance.14 A number of scholars have also explored
the relationship between innovative output and firm size utilizing various measures, most notably
patent and innovation counts, and these studies in general suggest that the number of patents or
innovations per R&D dollar declines with firm size (Acs and Audretsch 1988; Bound, Griliches
et al. 1984). Some studies further suggest that smaller firms actually account for a
disproportionately larger share of innovations relative to their size (Acs and Audretsch 1988;
Pavitt 1987).
Prior research in the pharmaceutical industry that examines the effects of scale on firms’
innovative performance is limited and also inconclusive. Early work by Comanor (1965) and
Vernon and Gusen (1974) find evidence of decreasing returns to scale in pharmaceutical R&D.
Later work confirms the lack of scale advantages with respect to drug research (Henderson and
Cockburn 1996) and drug development (Cockburn and Henderson 2001) at the drug program
level. By contrast, Dimasi et al. (1995), Schwartzman et al. (1996) and Arora et al. (2000)
suggest that there are economies of scale in pharmaceutical research.15
Considered together, the empirical evidence of scale effects in R&D overall and specific
to the pharmaceutical industry poses an interesting puzzle (Cohen and Klepper 1996). Why is it
14
For instance, an elasticity of R&D with respect to firm size greater than one does not imply an elasticity of
innovative output (e.g. patents or innovations) with respect to size greater than one.
15
Arora et al. (2000), however, do not allow for the identification of the specific aspects of development that yield scale
advantages nor do the authors identify the product areas in which scale effects are most likely present.
11
that large firms appear to conduct more R&D than smaller firms if they are comparatively less
efficient? One approach to shedding light on this question is to explore more directly the
incentives of firms of varying sizes within a particular industry to engage in R&D activities.
Implicit in the justifications for why larger firms may have advantages over smaller firms in
innovative activities is the assumption that there are direct and indirect cost advantages
associated with performing R&D in larger organizations, although few studies have attempted to
explore whether in fact these advantages exist. The lack of available micro-level data, moreover,
has generally forced researchers to explore scale advantages in R&D at a highly aggregated level
irrespective of product area or phase of R&D (Henderson and Cockburn 1996). To the extent that
there are (in)direct cost advantages for large firms performing innovative activities, these may be
tied to a given product area or the type of R&D activities being performed. As R&D often entails
significant trial and error, larger firms may be able to conduct more and more heterogeneous
experiments or specialize in particular scientific areas that guide technology options or phases of
development that speed commercialization in ways that smaller firms cannot. If critical inputs
can be spread over a larger base of more focused development activity, scale advantages may
result.
Firm-level scale benefits are likely to affect the costs of drug development noted above in
at least three distinct ways. First, drug development requires investment in substantial fixed
costs. To the extent that laboratories, testing facilities, computer resources and data analysis
technologies are able to be (re)deployed for multiple projects, firms may reap economies by
spreading the fixed costs of these resources over a larger base of development activities. These
advantages are likely to be particularly evident in late stage development projects (e.g. Phase
III), as late stage “confirmatory” projects tend to be much larger and more complex than earlier
“exploratory” projects, requiring organizations to make significantly larger investments in
infrastructure and support. Second, larger firms may be able to obtain scale economies through
their ability to attract and support highly specialized scientific and clinical personnel. Finally,
larger firms may be better positioned to make investments in specialized knowledge and
technologies related to given therapeutic and disease areas (e.g. specialized databases of patients
and investigators in a particular area, researchers and clinicians with relevant expertise,
specialized testing equipment and facilities, etc.).
12
The importance of phase and therapeutic area factors in conditioning the influence of
scale on performance provides an important justification for considering the nature of the
technology and the specific set of innovative activities being undertaken. These potential
differences suggest that the effects of scale will vary at the program level (Cockburn and
Henderson 2001; Henderson and Cockburn 1996), as well as between stages of innovation (e.g.,
research versus development) and stages of development (e.g., early stage versus late stage). Any
discussion of scale benefits must also necessarily explore the limits to scale, as the organizational
costs associated with performing R&D in larger organizations may eventually swamp any scale
benefits. We therefore examine the following hypothesis:
H2a:
Scale economies improve drug development, particularly in the later stages, but
beyond a certain point escalating coordination and agency costs lead to
diseconomies of scale.
Economies of scope are present if cost savings or performance benefits are realized when
two or more activities are conducted jointly in comparison to when these activities are conducted
separately (Panzar and Willig 1981). In the standard analysis of production, scope economies
result when activities can share productive inputs at little or no additional cost. Henderson and
Cockburn (1996) identify internal spillovers of knowledge as a second source of returns that
results from a more diverse R&D program. They argue that the knowledge developed and
accumulated in one R&D activity can be transferred to other R&D activities at little cost, but
with significant performance benefits. Some research suggests that beyond a certain point,
however, escalating coordination and agency costs eventually lead to diseconomies of scope
(Henderson and Cockburn 1996; Zenger 1994).
The most likely effects of greater scope economies on drug development arise from
knowledge spillovers, when the information generated in a given therapeutic or disease area
“spills over” and affects the costs and timing associated with developing new drugs in another
technological area. We distinguish and test for two types of spillovers in drug development.
First, firms that are more active in therapeutically-related groups develop knowledge that can be
more easily transferred to other related projects at little cost but with potentially significant
13
performance benefits.16 A more diverse development portfolio within a given technological area
may therefore allow firms to significantly decrease the time associated with developing new
drugs in that particular area. Firms that are active in multiple therapeutic areas—regardless of
“technological proximity”—may also reap performance benefits if the resources in one area are
relevant to a broad range of applications and can be utilized in any one of them without
diminishing their usefulness to other areas.17
Second, advantages in drug development may exist from broader functional scope. In
pharmaceuticals, the effective design and execution of the battery of clinical and non-clinical
tests that are required to successfully develop drug products requires sophisticated methods in
place to manage and communicate large volumes of data. The regulatory standards for getting
new drug applications approved by FDA have been steadily increasing over time and with them
the complexity and the amount of data that is generated throughout the drug development
process. Organizations must have access to, and be able to integrate, a broad range of business
functions, including toxicology, clinical medicine, product formulation, regulatory analysis and
process development (Spilker 1989). Organizations’ abilities to transfer general knowledge about
the effective management of their clinical trials across different functional areas, or to impart
medical knowledge from operating in more functional areas, may provide performance benefits
(Cockburn and Henderson 2001). The importance of functional breadth most likely increases as
drug products move into the latter and more complex stages of development. As Spilker notes,
“[t]he overriding principle of drug development is that activities may be planned, coordinated,
and to some degree controlled using appropriate systems” (1989: pp 19). Successful drug
development not surprisingly requires that firms bring together and effectively manage a broad
range of business functions (Spilker 1989). More functionally diverse firms may be able to
utilize development resources more cost-effectively in comparison to firms that are less
functionally diverse.
16
For example, knowledge generated in the development of anti-arrhythmic drugs within the cardiovascular therapeutic area
may spill over and allow for the easier development of targeted clinical protocols for a new hypertension treatment (also within
the cardiovascular area).
17
One example of this type of spillover cited by a number of interviewees relates to knowledge of certain aspects of the
regulatory process. For example, in 1992 the FDA promulgated a series of regulations outlining the requirements for firms
developing so-called "orphan drugs." The FDA's Orphan Drug program provides various incentives to firms that are developing
drugs for a host of rare diseases across a broad range of therapeutic areas. Interviewees suggested that as firms began acquiring
experience dealing with the FDA on orphan products, they were able to apply their knowledge to new orphan drugs, regardless of
therapeutic or disease area.
14
Firm performance associated with greater technological scope is likely to be greater in
earlier development projects than in later phases of work, as the importance of basic scientific or
disease-related knowledge is amplified in earlier stages of development (Azoulay 2002), in
contrast to later stage projects which tend to focus more on operational details. Furthermore, and
similar to scale economies, there may be limits to scope as the organizational costs associated
with performing R&D in very diverse technological or functional areas that eventually constrain
the benefits. We therefore examine the following hypothesis:
H2b:
Scope economies improve drug development, particularly in the earlier stages, but
beyond a certain point escalating coordination and agency costs lead to
diseconomies of scope.
4. ECONOMETRIC ANALYSIS
4.1. Data
Data for this paper were obtained from three sources. A proprietary data set providing
firm information and detailed project histories of drug development for over 500 CROs was first
obtained from DataEdge, a leading pharmaceutical consulting firm. This dataset contains
information on U.S. and European CROs of varying sizes and ownership structures (e.g., private,
public, university affiliated, government affiliated, etc.). Each of the organizations in the dataset
provided DataEdge with detailed project-level data, including the type of drug developed (e.g.
therapeutic area, disease group and drug indication), the specific tasks undertaken, the phase or
phases of development covered, specific start and end dates, and the number of subjects and test
centers involved.18
To ensure the integrity and the empirical validity of the data, a detailed review of industry
publications and promotional material for each of the CROs included in this study was then
performed. This information provided greater detail on individual CRO history, as well as the
specific types of information and technology services each of these CROs provided. We then
conducted a series of telephone and in-person interviews with a number of the firms in the
18
DataEdge also gathers detailed intra-firm information for each CRO, including firm age and size; the number of employees
and their functional backgrounds; geographic presence; specific hardware and software employed; the allocation of business
among major clients; and the types of facilities owned or leased.
15
dataset and with other pharmaceutical firms and CROs in order to reduce ambiguity and provide
greater detail around the factors that drive performance in drug development.
4.2. Econometric Approach
To examine the effects of the factors noted above on performance in drug development,
we utilize event history analysis. In particular, we employ a maximum likelihood estimation of a
parametric survival-time model with clustering to obtain robust variance estimates that adjust for
within-firm correlation to examine the likelihood that a development project is completed at time
t. The event we examine is the completion of a drug development project by a CRO. In the
analysis, each development project x is considered to be at risk of completion from firm i in each
time period t, until its completion occurs. To estimate drug development completion rates, we
use an exponential model in which there is no age parametric dependence specified in the
functional form of the model. This approach models the rate of a transition from an origin state
to a destination state as a function of the prescribed covariates. It takes the following general
form:
(
)
r jk = exp α jk 0 + A jk1α jk1 + A jk 2α jk 2 + ...
(1)
where rjk is the transition rate from origin state j (drug development project start) to destination
state k (project completion), with the observed covariate vector Ajk and parameters to be
estimated αjk and constant αjk0. The duration of an event is modeled using an exponential
distribution, and the relationship between the covariates and the transition rate is specified as
log-linear to ensure transition rate estimates are non-negative. Significant positive (negative)
coefficient estimates indicate drug development project completion rates increase (decrease)
when the associated covariate increases (decreases) in value.
A central critique of empirical work within the economics of R&D has been the frequent
use of highly aggregated data to proxy for the variables of interest. To address this criticism, we
construct detailed firm- and project-level data using the DataEdge database. To ensure that we
are not comparing performance across different types of projects, we select only those clinical
development projects in which the management of the trials themselves was outsourced to
CROs. We also select only those drug development projects that were completed by CROs to
avoid problems associated with censoring. Our sample includes clinical projects covering the
16
four major phases of development in order to explore the roles of experience and scale and scope
economies in early versus late stage drug development. Finally, the drug development projects
cover a variety of different therapeutic and disease areas.
4.3. Dependent and Independent Variables
The dependent variable (TIME) corresponds to the fractional number of months from the
date a given drug development project was initiated by a CRO to its completion date. Although it
is possible to identify other important performance measures, time to complete is an important
measure for CROs in particular and in pharmaceutical new product development
pharmaceuticals in general, especially in consideration to the substantial competitive and
financial costs associated with delay.
Firm Experience – Experiential learning is hypothesized as having two components. First, may
develop unique skill sets and knowledge bases working in specific technological areas. The
pharmaceuticals industry fortunately employs a fairly standard taxonomy into which drug
products can be assigned (i.e., major therapeutic area and sub-areas, or drug indications, falling
under each). We create two variables that measure CRO experience at each level within this
taxonomy. TA_EXP measures CROs’ prior experience in each therapeutic area. Specifically,
experience in therapeutic area i at time t is determined by:
TA _ EXPi , t = δTA _ EXPi , t −1 + TAi , t
( 2)
where TA_EXPi,t represents a count of prior drug development projects for a given CRO in
therapeutic category i at period t, and TAi,t represents a count of current drug development
projects in therapeutic category i at period t. The depreciation factor (δ) allows recent experience
to be weighted more heavily than past experience, which is consistent with the notion of
organizational forgetting (Argote 1999; Benkard 1999).19 For a given drug development project,
t is defined as the date that the product entered clinical trials. A similarly depreciated experience
variable (IND_EXP) is developed for drug indications within a therapeutic area. The taxonomy
allows for a determination of whether experience matters more within the more broadly defined
therapeutic area or within the more narrowly defined drug indication, as well as for a
19
We vary this variable from 10 to 30 per cent in the econometric analysis, and the results are consistent.
17
determination of any differences that exist between the experience variables according to the
particular phase of the clinical trial.
Second, firms may develop greater experience operating in particular phases of drug
development. Some CROs may be more proficient in the earlier or more knowledge-intensive
stages of drug development (Azoulay 2002), while others may develop greater experiential
learning in the latter clinical trial stages, which tend to be more data management intensive. To
test for this possibility, PHASE_EXP measures CROs’ prior experience in the phase of drug
development for which the current drug development project is for. Specifically, firm experience
in clinical phase i at time t is given by:
PHASE _ EXPi , t = δPHASE _ EXPi , t −1 + PHASE i , t
(3)
where PHASE_EXPi,t represents a count of prior drug development projects for a given CRO in
clinical phase i at period t, and PHASEi,t represents the ongoing drug development projects in
clinical phase i at period t.
Scale and Scope Economies – The variable SCALE represents a count of the number of full-time
staff at the time a drug development project is undertaken for a given CRO, and is our measure
primary of scale economies.20 To explore knowledge spillovers within related technology areas,
a technology “area” must be defined. The use of the therapeutic area—drug indication taxonomy
fortunately reduces the difficulty of this problem. We utilize two measures of “technology area”
scope and one measure of “functional area” scope in the empirical analysis. TA_SCOPE
measures CROs’ therapeutic area scope, or what we consider the broadest measure of CROs’
development portfolio scope. This variable is defined as a count of the number of therapeutic
areas in which a CRO is active in and is measured in the year a given drug development project
is undertaken. Similarly, IND_SCOPE represents a count of the number of drug indications
within the same therapeutic area that the CRO is active. Finally, FUNC_SCOPE measures
CROs’ functional scope, or the number of functional areas in which CROs have at least twenty
employees.21
20
Two other measures of size—R&D spending and revenue—were rejected because of accuracy questions and anomalies in
reporting.
21
The numbers of employees per functional area are provided to DataEdge by CROs through questionnaires. The
classification scheme creates the following functional areas: (1) clinical, (2) data management, (3) laboratory, (4) medical, (5)
18
Control Variables – In addition the independent variables defined above, a number of other
control measures are included in this econometric analysis. We control for the effect of
therapeutic area and the phase of development work being undertaken in a given drug
development project in order to capture any time-invariant dimensions that are related to
development phase or therapeutic area that are not explained by the independent variables or
other control variables. Both of these factors have been identified in the previous literature as
having a significant effect on drug development (DiMasi 1999; DiMasi, Hansen et al. 1991). We
also include firm fixed effects in the specification to capture all time-invariant dimensions of the
CROs in the sample.
Beyond therapeutic, development phase and firm fixed effects, the sheer size of a drug
development project should affect drug development project time to completion. To control for
project size considerations, we employ CENTERS and SUBJECTS, which represent the number
of clinical study centers and patients involved in a given drug development project, respectively.
We expect that the number of clinical study centers to decrease development project time due to
scale economies, while the number of subjects to have the opposite effect due to the managerial
burdens created. Finally, the novelty of a particular drug may influence the time required to
perform drug development activities. With less pre-existing knowledge regarding the therapeutic
characteristics of a drug, firms must expend extra effort profiling and exploring its clinical effect,
ceteris paribus. NOVELTY therefore measures the number of marketed drug products within a
particular therapeutic category.
The key variables defined above are summarized in Table 1 along with their expected
influence on the underlying hazard rate specified in the econometric model. As our measures
represents the hazard of a drug development project being completed at a given time t, variables
that lead to shorter (longer) drug development projects have positive (negative) coefficients.
Descriptive statistics for all of the variables and correlation statistics for the independent
variables are presented in Tables 2 and 3, respectively.
quality assurance, and (6) regulatory. Two potential problems with this classification are (1) whether twenty individuals is a
sufficiently accurate proxy and (2) whether the areas are overly narrow and the arbitrary assignment of employees to these areas.
To the first measurement problem, sensitivity analysis in the econometric analysis indicates that there are no significant
differences in using five individuals as the cutoff value. To the second, interviews suggest that CRO employees frequently wear
multiple functional hats. These critiques notwithstanding, the DataEdge classification scheme covers the most obvious functional
areas associated with drug development, providing at least a first approximation of firms’ functional scope.
19
5. EMPIRICAL RESULTS
5.1. Baseline Analysis
The econometric results of firm experience and scale and scope economies and their
relation to drug development project performance are presented in Tables 4-6. Table 4 presents
the baseline results of the empirical analysis, while Table 5 provides an in-depth examination of
scale and scope economies and Table 6 examines drug development by clinical phase. All of the
models presented easily reject likelihood ratio null hypothesis tests for the inclusion of the fixed
effect, control and independent variables, at least at the .001 level.
Model (1) of Table 4 presents the baseline results. The baseline hazard rate is 0.040,
which translates into a probability that a development project will be completed in a given six
month period of roughly 25 per cent. The control variables are significant in some models and
generally in the expected direction. An increase in the number of subjects involved in a drug
development project leads to a small increase in time to completion, while the number of study
centers involved decreases development time completion. Both of these results are consistent
with earlier work in the pharmacoeconomics literature (Kaitin and Manocchia 1997; Kaitin,
Manocchia et al. 1994).
Model (2) introduces the experience variables into the econometric analysis. We argue
that greater “technological area” experience and clinical phase experience improve drug
development project completion time. The results demonstrate that greater experience within a
particular therapeutic area significantly improves performance, but greater experience within a
drug indication area does not. We surmise that greater therapeutic area experience provides
abundant and useful technological area knowledge via experiential learning by doing, whereas
greater drug indication experience is either too narrowly focused or more important in particular
phases of drug development—a possibility that we test for below. The results do not demonstrate
that greater clinical phase experience significantly improves development performance.
Model (3) replaces the experience variables with measures of scale and scope. Scale
economies are not found, failing to support hypothesis H3a. However, both of the technological
area scope variables are significant, but in opposite directions. In particular, greater therapeutic
area scope significantly increases drug development time while greater drug indication scope
(within a particular therapeutic area) reduces drug development time. We therefore argue that
20
knowledge spillovers exist within a therapeutic area, but become more limited between or across
significantly diverse therapeutic areas. The functional scope variable is also significant, which
provides support for hypothesis H3b and indicates that CROs who offer broad functional
services realize performance efficiencies. Model (4) includes both the experience and scale and
scope variables, and the results obtained are broadly consistent with the earlier models. In
particular, the therapeutic area experience variable and the technological area and functional
scope variables maintain statistical significance. We examine the economic significance of these
results in the discussion section below.
5.2. Scale and Scope Economy Analysis
Table 5 presents the results of a more in-depth analysis of drug development scale and
scope economies. Model (1) repeats the Model (4) results of Table 4 solely for comparison
purposes. Model (2) examines scale economies, while Models (3)-(5) examine scope economies
in therapeutic area, drug indication and function, respectively. No evidence of scale effects are
found in Model (2), indicating no advantage to size in the speed with which drug development
projects can be completed.
Model (3) indicates substantial scope diseconomies in therapeutic area, which suggests
that overly broad scope—in terms of the number of therapeutic areas a CRO is active in—
hinders development performance. These results obtain possibly because knowledge spillovers
between diffuse technological areas are limited or because CRO resources are either spread too
thin or have limited abilities to develop adequate knowledge bases within diffuse technological
areas. Model (4) replaces the squared therapeutic area term with a squared drug indication term.
The results indicate that greater drug indication scope provides significant performance benefits,
but eventually leads to diseconomies. Finally, Model (5) tests for scope effects according
function, and the results largely support this argument.
Table 5 provides fairly robust support for scope economies in technological and
functional areas, but not for scale economies. Although these findings are consistent with other
research that examines drug development performance (Cockburn and Henderson 2001), our
findings also indicate that scope economies result from greater diversity within a therapeutic area
(e.g., drug indications) and not between or among therapeutic areas. In particular, overly broad
scope—in terms of therapeutic areas—appears to lead to performance limitations, while broad
21
scope within a particular therapeutic area—in terms of drug indications—creates performance
benefits. We examine these results in more detail in the discussion section below.
5.3. Clinical Phase Analysis
Sample size limitations unfortunately prevent an examination of each clinical phase
independently. Model (1) of Table 6 therefore examines the effects of experience and scale and
scope in Phase I and Phase II combined—considered the more knowledge-intensive clinical
phases—while Model (2) presents an analysis of Phases III and IV combined—considered the
more data-intensive phases. The Model (1) results demonstrate the importance of scale and drug
novelty in early-phase development. In particular, the number of clinical centers involved and
the number of drug products available within a therapeutic area have significant influences on
drug development project completion performance. Greater firm experience and scale and scope
economies fail to reach statistical significance. Given that Phase I and Phase II tend to be more
knowledge-intensive in comparison to the later-stage phases, these results are somewhat
surprising, but may be due to the relatively small sample size.
The Model (2) results are as predicted and broadly consistent with those obtained in
Tables 4 and 5. In particular, the importance of greater therapeutic area experience and scope
economies in drug indication and scope diseconomies in therapeutic area are realized.
5.4. Discussion
The results presented in the tables above provide support for the importance of firm
experience and scope economies on drug development performance. With respect to experiential
learning, we find that greater therapeutic area experience generally yields drug development
performance advantages, while greater drug indication experience does not. There is some
evidence that experience is also important in more knowledge-intensive development activities.
While it is not surprising that more experience helps to improve performance, given the past
research that examines experiential learning, we importantly find that the specific kind of
experience matters. In particular, experience that provides abundant and useful technological
area knowledge that is neither “too broad” nor “too focused” is important. Concentrating in a
single sub-area—evidenced by experience in a particular drug indication—may pigeonhole
organizations in the things it can (and cannot) do and limit the number or variety of future
projects that it can take on. Table 7 illustrates the economic effect of greater therapeutic area
22
experience on drug development utilizing its calculated hazard ratio. Every additional year of
related therapeutic area experience for the current drug development project increases the hazard
by more than ten per cent. The mean level of therapeutic experience increases the hazard by a
factor of 1.3, while an additional standard deviation increase from the mean level increases the
hazard by a factor of more than four. We also find some evidence of experiential benefits in
more knowledge-intensive development activities.
Our results do not confirm any scale advantage in drug development for CROs. In
addition, there appears to be limited effects from size in early versus late stages of development.
We do find significant scope economies in drug development, however. In particular, firms who
develop a limited number of potentially overlapping knowledge bases within one or a few
therapeutic areas, as opposed to developing knowledge bases in multiple—and possibly
unrelated—therapeutic areas achieve performance benefits. Working in a large number of
therapeutic areas potentially overextends firms or limits their abilities to develop sufficient
knowledge, possibly because spillovers are limited. More related knowledge development
activities—as evidenced by the scope of current drug indication projects within a specific
therapeutic area—provides significant performance improvement opportunities, again because of
the knowledge spillovers that likely exist within single technological area. Our results therefore
extend the work of Henderson and Cockburn (1996) and others (Chandler 1992; Teece 1980)
who suggest that one of the most important determinants of scope in innovative intensive
industries is the opportunity to take advantage of intra-firm spillovers of knowledge within
closely related areas of technological knowledge. In addition to these knowledge-related scope
effects, we find evidence that functional scope is important for firms particularly in later stage
clinical trials.
Table 7 illustrates the economic effects of the various measures of scope economies on
drug development utilizing the respective calculated hazard ratios. For instance, a one unit
increase and one standard deviation increase in therapeutic area scope decrease the hazard by a
factor of 0.11 and 2.04, respectively. By contrast, an increase in drug indication scope equal to
its mean increases the hazard by a factor of more than seventeen. Severe completion time
penalties therefore accrue to those firms with overly broad scope, while benefits exist for those
firms with significant knowledge within particular targeted areas.
23
Our results suggest that the scope of innovative activities provide firms with significant
performance advantages in conducting drug development. Efforts to capitalize on these cost
advantages may help to explain many of the noticeable changes taking place in the
pharmaceutical industry. In particular, both the pharmaceutical and contract research markets
have witnessed an intense wave of consolidation in recent years. In the CRO market, this
consolidation has led to the emergence of large, functionally and therapeutically diversified
firms. Our findings suggest that there is some quantitative justification for this industrial
reorganization.
A number of caveats must be noted to the results discussed above. First, we explore only
projects that were conducted by CROs. As such, our results are at the mercy to any biases that
may be present in the selection of drug development projects by pharmaceutical firms for
outsourcing. Second, we do not control for the inter-firm spillovers that may be present and have
been found to be important drivers of innovative performance in prior research. Third, our
interviews suggest that there are important differences between small and large CROs with
respect to performance. In particular, larger firms tend to outperform smaller firms on large-scale
data-intensive projects, yet there are gains from specialization that may advantage smaller firms.
Although our results regarding size effects do not support these claims, we believe that more
qualitative and quantitative work is necessary. Finally, all of our results are subject to the
qualification that drug development project time-to-complete is but one measure of performance
in the pharmaceuticals industry. To be sure, efforts to explore many of the factors examined
using other measures would potentially clarify our results.
6. Conclusion
This paper provides a detailed look at the importance of firm experience and scale and
scope on development performance in an important and highly-innovative setting. The analysis
presented provides a number of insights into the importance and interaction of firm experience
and scale and scope in the new product development process. In doing so, the paper adds to the
literature that examines the economics of R&D and innovative performance. Prior literature in
the economics of R&D literature has hinted at the importance of scale and scope in development
without exploring how the stage of development or firm experience in particular technological
areas may condition the magnitude of these effects. One central reason for these problems is that
24
many of the factors that are central to firm performance are not observable in publicly available,
aggregate data (Rouse and Dallenbach 1999). This paper overcomes this limitation by utilizing a
proprietary data set containing detailed project and firm-level information to more accurately
measure the factors that condition the time required to perform drug development.
Our empirical results underscore the importance of experiential learning and scale and
scope economies in differentiating firm innovative performance. The results also suggest that
some of these factors are more important in particular stages of development than others. These
findings importantly denote that both experiential learning and scale and scope help to explain
performance differences in research and development and in innovation. To date, few researchers
(save Henderson and Cockburn’s (1996) examination of drug discovery) have explored how the
size and shape of firms’ research portfolio, along with greater experience, affects innovative
performance. Our results extend this approach by finding that an important determinant within
innovative intensive industries is taking advantage of intra-firm spillovers of knowledge within
closely related areas of technological knowledge.
We also believe that our results can easily be extended to other settings and industries. In
particular, we expect that settings or industries in which firms make different but related
products using different manufacturing processes possess intra-firm knowledge spillovers.
Appropriate industry setting examples are likely to include, but not be limited to chemicals,
consumer electronics, semiconductor products, and software. In each setting, experience and
knowledge within a particular technological area is likely to readily translate into improving
performance in other closely related technological areas.
25
REFERENCES
Acs ZJ and Audretsch DB. 1988. Innovation in Large and Small Firms. American Economic
Review 78: 678-90.
Argote L. 1999. Organizational Learning: Creating, Retaining and Transferring Knowledge.
Boston, Kluwer Academic Publishers.
Arora A and Gambardella A. 1994. The Changing Technology of Technological Change:
General and Abstract Knowledge and the Division of Innovative Labor. Research Policy
23(5): 523-532.
Arora A, Gambardella A, Pammolli F and Riccaboni M (2000). The Nature and Extent of the
Market for Technology in Biopharmaceuticals. Technological Policy and Innovation:
Economic and Historical Perspectives, Paris France.
Azoulay P. 2002. The Many Faces of Outsourcing: Adjustment Costs, Transaction Costs, and
Governance Spillovers. Columbia Graduate School of Business Working Paper.
Baum JAC, Li SX and Usher JM. 2000. Making the Next Move: How Experiential and Vicarious
Learning Shape the Locations of Chains' Acquisitions. Administrative Science Quarterly
45: 766-801.
Benkard LC. 1999. Learning and Forgetting: The Dynamics of Aircraft Production. Palo Alto,
CA, Stanford University, Graduate School of Business.
Bogner W and Thomas H. 1994. Core Competence and Competitive Advantage: A Model and
Illustrative Evidence from the Pharmaceutical Industry. in G Hamel and A Heene, eds.,
Competence-Based Competition. New York, John Wiley and Sons, Ltd.
Bound JC, Griliches Z, Hall BH and Jaffee A. 1984. Who Does R&D and Who Patents? in Z
Griliches, ed. R&D Patents and Productivity. Chicago, University of Chicago Press for
the NBER.
Chandler AD. 1992. Organizational Capabilities and the Economic History of the Industrial
Enterprise. Journal of Economic Perspectives 6(3): 79-100.
Christensen CM and Bower JL. 1996. Customer Power, Streatic Investment and the Failure of
Leading Firms. Strategic Management Journal 17(197-218).
CMR_International. 1999. R&D Briefing: Outsourcing in Operation - Contracting out
Pharmaceutical R&D. Surrey. United Kingdom, CMR International.
Cockburn I and Henderson R. 1994. Racing to Invest? The Dynamics of Competition in Ethical
Drug Discovery. Journal of Economics and Management Strategy 3(3): 481-19.
Cockburn I and Henderson R. 1999. The Diffusion of Science Driven Drug Discovery:
Organizational Change in Pharmaceutical Research. National Bureau of Economic
Research. Cambridge, MA.
Cockburn I, Henderson R and Stern S. 1999. Balancing Incentives: The Tension between Basic
and Applied Research. Cambridge, MA., NBER.
26
Cockburn I, Henderson R and Stern S. 2000. Untangling the Origins of Competitive Advantage.
Strategic Management Journal 21: 1123-1145.
Cockburn IM and Henderson RM. 2001. Scale and Scope in Drug Development: Unpacking the
Advantage of Size in Pharmaceutical Research. Journal of Health Economics 20: 10331057.
Cohen W. 1995. Empirical Studies of Innovative Activity. in P Stoneman, ed. Handbook of the
Economics of Innovation and Technological Change. Oxford, Blackwell Publishers: 182264.
Cohen WM and Klepper S. 1996. A Reprise of Size and R&D. The Economic Journal 106(437):
925-951.
Cohen WM and Levin RC. 1989. Empirical Studies of Innovation and Market Structure. in R
Schmalensee and RD Willig, eds., Handbook of Industrial Organization. Amsterdam,
Elsevier Science Publishers: 1060-1107.
Comanor WS. 1965. Research and Technical Change in the Pharmaceutical Industry.". Review of
Economics and Statistics 47: 182-190.
Comanor WS. 1967. Market Structure, Product Differentiation, and Industrial Research.
Quarterly Journal of Economics 81: 639-57.
Delios A and Henisz WJ. 2000. Japanese Firms' Investment Strategies in Emerging Economies.
Academy of Management Journal 43(3): 305-323.
DiMasi J. 1999. New Drug Innovation and Pharmaceutical Industry Structure: Trends in the
Output of Pharmaceutical Firms.
DiMasi J, Grabowski H and Vernon J. 1995. R&D Costs, Innovative Output and Firm Size in the
Pharmaceutical Industry. International Journal of the Economics of Business 2: 201-219.
DiMasi J, Hansen R, Grabowski H and Lasagna L. 1991. Cost of Innovation in the
Pharmaceutical Industry. Journal of Health Economics 10: 107-142.
Fisher FM and Temin P. 1973. Returns to Scale in Research and Development: What Does the
Schumpeterian Hypothesis Imply? Journal of Political Economy 81: 56-70.
Galbraith JK. 1952. American Capitalism: The Concept of Countervailing Power. Boston,
Houghton Mifflin.
Gambardella A. 1995. Science and Innovation: The Us Pharmaceutical Industry During the
1980s. Cambridge, UK, Cambridge University Press.
Grabowski H and Vernon M. 1990. A New Look at the Returns and Risks to Pharmaceutical
R&D. Management Science 36(7): 804-82.
Griliches Z. 1995. R&D Productivity: Econometric Results and Measurement Ideas. in P
Stoneman, ed. Handbook of the Economics of Innovation and Technological Change.
Oxford, Blackwell Publishers Ltd.: 52-89.
Haleblian J and Finklestein S. 1999. The Influence of Organization Acquisition Experience on
Acquisition Performance: A Behavioral Learning Theory Perspective. Administrative
Science Quarterly 44: 29-56.
27
Henderson R. 1994. The Evolution of Integrative Capability: Innovation in Cardiovascular Drug
Discovery. Industrial and Corporate Change 3(3): 607-630.
Henderson R and Cockburn I. 1994. Measuring Competence? Exploring Firm Effects in
Pharmaceutical Research. Strategic Management Journal 15: 63-84.
Henderson R and Cockburn I. 1996. Scale, Scope and Spillovers: The Determinants of Research
Productivity in Drug Discovery. RAND Journal of Economics 27(1): 32-59.
Henisz WJ and Delios A. 2001. Uncertainty, Imitation, and Plant Location: Japanese
Multinational Corporations, 1990-1996. Administrative Science Quarterly 46(3): 443475.
Kaitin K and Manocchia M. 1997. The New Drug Approvals of 1993, 1994, and 1995: Trends in
Drug Development. American Journal of Therapeutics 4: 46-54.
Kaitin K, Manocchia M, Seibring M and Lasagna L. 1994. The New Drug Approvals of 1990,
1991, and 1992: Trends in Drug Development. Journal of Clinical Pharmacology 34:
120-127.
Leonard-Barton D. 1992. Core Capabilities and Core Rigidities: A Paradox in Managing New
Product Development. Strategic Management Journal 13: 111-125.
Levinthal DA and March JG. 1993. The Myopia of Learning. Strategic Management Journal 14:
1495-112.
Malerba F and Orsenigo L. 2001. Innovation and Market Structure in the Dynamics of the
Pharmaceutical Industry and Biotechnology: Towards a History Friendly Model.
Aalborg, Denmark.
March J. 1991. Exploration and Exploitation in Organizational Learning. Organizational Science
2(1): 71-87.
March JG. 1988. Decisions in Organizations. New York, Blackwell.
March JG and Simon H. 1958. Organizations. New York, John Wiley.
Mathieu M. 1997. New Drug Development: A Regulatory Overview. Waltham, MA, Parexel
International corporation.
Mitchell W, Shaver JM and Yeung B. 1992. Getting There in a Global Industry: Impacts on
Performance of Changing International Presence. Strategic Management Journal 13(6):
419-432.
Nelson RR, Peck MJ and Kalachek ED. 1967. Technology, Economic Growth and Public Policy.
Washington, DC, The Brookings Institution.
Nelson RR and Winter SG. 1982. An Evolutionary Theory of Economic Change. Cambridge,
Belknap Press.
O.T.A. US. 1984. Nuclear Power in an Age of Uncertainty, Washington DC:GPO. United States,
President's Commission on the Accident at Three Mile Island.
OTA. 1993. Pharmaceutical R&D: Costs, Risks and Rewards. Washington, DC, Office of
Technology Assessment.
28
Panzar JC and Willig RD. 1981. Economies of Scope. American Economic Review 71(2): 268272.
Patel P and Pavitt K. 1995. Patterns of Technological Activity: Their Measurement and
Interpretation. in P Stoneman, ed. Handbook of the Economics of Innovation and
Technological Change. Cambridge, Blackwell Publishers Ltd.: 14-51.
Pavitt K. 1987. The Objectives of Technology Policy. Science and Public Policy 14: 182-188.
Pennings JM and Harianto F. 1992. The Diffusion of Technological Innovation in the
Commercial Banking Industry. Strategic Management Journal 13: 29-46.
PhRMA. 1999. Industry Profile. Washington, DC, Pharmaceutical Manufacturers Association.
Podolny JM. 1994. Market Uncertainty and the Social Character of Economic Exchange.
Administrative Science Quarterly 39(3): 458(26).
Rouse MJ and Dallenbach US. 1999. Rethinking Research Methods for the Resource-Based
Perspective: Isolating Resources of Sustainable Competitive Advantage. Strategic
Management Journal 20(5): 487-494.
Sah RJ and Stiglitz JE. 1988. Committees, Hierarchies and Polyarchies. Economic Journal 98:
451-70.
Scherer FM. 1965. Firm Size, Market Structure, Opportunity, and the Output of Patented
Inventions. American Economic Review 55: 1097-1125.
Scherer FM. 1980. Industrial Market Structure and Market Performance. Chicago, Rand
McNally College Pub.
Scherer FM and Ross D. 1990. Industrial Market Structure and Economic Performance. Boston,
Houghton Mifflin Company.
Schumpeter J. 1951. The Theory of Economic Development. Cambridge, MA., Harvard
University Press.
Schumpeter JA. 1942. Capitalism, Socialism and Democracy. London, G. Allen & Unwin Ltd.
Schwartzman D and Cognato A. 1996. Has Pharmaceutical Research Become More Scientific?
Review of Industrial Organization 11: 841-51.
Spilker B. 1989. Multinational Drug Companies: Issues in Drug Discovery and Development.
New York, Raven Press.
Stoneman P. 1995. Handbook of the Economics of Innovation and Technological Change.
Oxford, UK and, Blackwell Publishers Ltd.
Stuart TE and Podolny JM. 1996. Local Search and the Evolution of Technological Capabilities.
Strategic Management Journal 17(Summer Special Issue): 21-38.
Teece DJ. 1980. Economies of Scope and the Scope of the Enterprise. Journal of Economic
Behavior and Organization 1: 223-247.
Teece DJ. 1982. Towards an Economic Theory of the Multiproduct Firm. Journal of Economic
Behavior and Organization 3: 39-63.
29
Vernon JM and Gusen P. 1974. Technical Change and Firm Size: The Pharmaceutical Industry.
Review of Economics and Statistics 56: 223-47.
Wernerfelt B. 1984. A Resource-Based View of the Firm. Strategic Management Journal 12: 7594.
Wiggins S. 1981. Product Quality Regulation and New Drug Introductions: Some New Evidence
from the 1970s. Review of Economics and Statistics 63(4): 615-619.
Zenger TR. 1994. Understanding Organizational Diseconomies of Scale: The Allocation of
Engineering Talent, Ideas, and Effort by Firm Size. Management Science 40: 708-729.
30
Table 1: Dependent and Independent Variables
Name
Units
Description
Predicted
Effect
DEPENDENT VARIABLE
TIME
#
Fractional number of months from the start to completion of a
given drug development project
INDEPENDENT VARIABLES
#
Discounted therapeutic area experience
TA_EXP
#
Discounted drug indication experience
IND_EXP
#
Discounted experience in particular clinical phase
PHASE_EXP
SCALE
TA_SCOPE
#
#
IND_SCOPE
#
FUNC_SCOPE
0…6
Number of full-time staff members employed by the CRO.
Number of therapeutic areas in which the CRO has project
experience
Number of indications within a given disease group in which
the CRO has project experience
Measure of functional scope. Number of functional areas the
CRO has at least twenty employees
CONTROL VARIABLES
#
Number of subjects involved in a given drug development
SUBJECT
project
#
Number of centers involved in a given drug development
CENTERS
project
#
Number of marketed products in a given therapeutic area
NOVELTY
THER. AREA
PHASE
FIRM
Therapeutic area fixed effect
Development phase fixed effect
Firm fixed effect
31
+
+
+
+
+
+
+
−
+
−
Table 2: Descriptive Statistics
Variable
N
Mean
S.D.
Min
Max
Therapeutic Areas:
Cardiovascular
Gastrointestinal
Central Nervous System
Anti-Infective
Oncology
Immuno-Modulation/Anti-Inflammatory
Dermatology
Endocrine
Pharmacokinetics
Hematology
Ophthalmology
Genitourinary System
Respiratory System
Pain and Anesthesia
129
129
129
129
129
129
129
129
129
129
129
129
129
129
0.09
0.04
0.14
0.14
0.06
0.06
0.06
0.02
0.06
0.14
0.02
0.01
0.04
0.14
0.28
0.19
0.34
0.34
0.23
0.23
0.23
0.15
0.25
0.35
0.15
0.08
0.34
0.34
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
1
1
1
1
1
1
1
1
1
1
1
1
Development Phase
Phase 1
Phase 2
Phase 3
Phase 4
129
129
129
129
0.18
0.11
0.53
0.18
0.38
0.31
0.50
0.39
0.0
0.0
0.0
0.0
1
1
1
1
Dependent Variable:
TIME
129
24.84
12.94
0.3
72.0
Independent Variables:
SUBJECT
CENTERS
NOVELTY
TA_EXP
IND_EXP
PHASE_EXP
SCALE
TA_SCOPE
IND_SCOPE
FUNC_SCOPE
129
129
129
129
129
129
129
129
129
129
593.33
79.97
9.50
1.21
0.39
12.85
396.91
6.67
16.69
1.53
1636.90
333.17
10.43
2.83
1.05
17.00
954.18
3.24
18.26
1.51
0.0
0.0
1.0
0.0
0.0
0.0
14.0
1.0
1.0
0.0
16000.0
3000.0
51.0
24.0
8.0
90.0
3602.0
14.0
74.0
6.0
32
CENTERS
NOVELTY
TA_EXP
IND_EXP
PHASE_EXP
SCALE
TA_SCOPE
IND_SCOPE
1.000
0.105
0.012
-0.088
-0.3108
-0.2052
0.2712
-0.111
0.013
-0.061
-0.1924
1.000
0.9157
0.114
-0.068
-0.040
0.027
0.086
0.033
0.059
0.145
1.000
0.2617
-0.060
-0.013
0.031
-0.017
-0.055
-0.053
0.076
1.000
0.065
0.1893
0.055
-0.022
0.149
0.008
0.058
1.000
0.7675
-0.156
0.4535
0.3932
0.4738
0.4599
1.000
-0.120
0.3034
0.2916
0.3032
0.3456
1.000
-0.104
-0.034
-0.039
-0.2069
1.000
0.6832
0.6140
0.6041
1.000
0.8496
0.6143
1.000
0.7945
Bold indicates pair-wise correlation significance at 0.05 level.
33
FUNC_SCOPE
SUBJECT
TIME
SUBJECT
CENTERS
NOVELTY
TA_EXP
IND_EXP
PHASE_EXP
SCALE
TA_SCOPE
IND_SCOPE
FUNC_SCOPE
TIME
Table 3: Correlations
1.000
Table 4: Baseline Results
(DV = Drug development project completion TIME)
(1)
Variable
CONSTANT
β
(SE)
-3.3872
(0.5584)
(2)
P>|z|
0.000
β
(SE)
-3.2626
(0.5543)
(3)
P>|z|
0.000
β
(SE)
-2.7802
(0.4488)
(4)
P>|z|
0.000
β
(SE)
-2.7288
(0.4648)
P>|z|
0.000
CONTROL
SUBJECT
CENTERS
NOVELTY
-0.0001
(0.0001)
0.0005
(0.0003)
-0.0064
(0.0105)
0.124
0.077
0.543
-0.0001
(0.0001)
0.0005
(0.0003)
-0.0064
(0.0105)
0.124
0.077
0.543
-0.0001
(0.0001)
0.0005
(0.0003)
0.0044
(0.0095)
0.136
0.120
0.643
-0.0001
(0.0001)
0.0004
(0.0002)
-0.0002
(0.0100)
0.157
0.128
0.986
EXPERIENCE
TA_EXP
0.1195
(0.0265)
-0.0646
(0.0704)
-0.0027
(0.0033)
IND_EXP
PHASE_EXP
0.0965
(0.0245)
-0.0367
(0.0693)
0.0000
(0.0034)
0.000
0.359
0.416
0.000
0.596
0.996
ECONOMIES
SCALE
-0.0002
(0.0002)
-0.1314
(0.0248)
0.0292
(0.0106)
0.1445
(0.0677)
TA_SCOPE
IND_SCOPE
FUNC_SCOPE
0.280
0.000
0.006
0.033
-0.0002
(0.0002)
-0.1207
(0.0240)
0.0225
(0.0122)
0.1203
(0.0641)
0.382
0.000
0.066
0.060
FIXED EFFECTS
THER. AREA
Inc.
Inc.
Inc.
Inc.
PHASE
Inc.
Inc.
Inc.
Inc.
FIRM
Inc.
Inc.
Inc.
Inc.
N
Log likelihood
LR Test (#vars)
LR Test (R-U)
129
129
129
129
-495.42
-480.73
-475.23
-465.28
23.76
0.205
53.13
0.000
64.14
0.000
84.04
0.000
29.37
0.000
40.37
0.000
60.28
0.000
Table 5: Scale and Scope Results
(DV = Drug development project completion TIME)
(1)
β
(SE)
-2.7288
(0.4648)
Variable
CONSTANT
(2)
P>|z|
0.000
β
(SE)
-2.6475
(0.4821)
(3)
P>|z|
0.000
β
(SE)
-3.0908
(0.5978)
(4)
P>|z|
0.000
β
(SE)
-2.7113
(0.4573)
(5)
P>|z|
0.000
β
(SE)
-2.6440
(0.4750)
P>|z|
0.000
CONTROL
SUBJECT
-0.0001
(0.0001)
0.0004
(0.0002)
-0.0002
(0.0100)
CENTERS
NOVELTY
0.157
0.128
0.986
-0.0001
(0.0001)
0.0004
(0.0002)
0.0007
(0.0099)
0.147
0.116
0.940
-0.0001
(0.0001)
0.0003
(0.0002)
0.0020
(0.0104)
0.193
0.157
0.844
-0.0001
(0.0001)
0.0004
(0.0002)
0.0011
(0.0103)
0.147
0.115
0.915
-0.0001
(0.0001)
0.0004
(0.0002)
0.0011
(0.0099)
0.154
0.120
0.908
EXPERIENCE
TA_EXP
0.0965
(0.0245)
-0.0367
(0.0693)
0.0000
(0.0034)
IND_EXP
PHASE_EXP
0.000
0.596
0.996
0.1003
(0.0218)
-0.0417
(0.0653)
0.0002
(0.0033)
0.948
0.0944
(0.0240)
-0.0291
(0.0686)
-0.0012
(0.0038)
0.760
-0.0002
(0.0002)
0.000
0.523
0.746
0.0977
(0.0240)
-0.0369
(0.0684)
-0.0002
(0.0035)
0.497
0.735
0.000
0.672
0.958
0.1006
(0.0220)
-0.0394
(0.0665)
0.0003
(0.0033)
0.0000
(0.0003)
0.953
0.0002
(0.0003)
0.613
-0.1349
(0.0211)
0.000
-0.1345
(0.0217)
0.000
0.013
0.0251
(0.0108)
0.021
0.000
0.590
0.000
0.554
0.925
ECONOMIES
SCALE
-0.0002
(0.0002)
SCALE2
TA_SCOPE
-0.1207
(0.0240)
TA_SCOPE2
IND_SCOPE
0.0225
(0.0122)
0.000
0.0001
(0.0004)
0.0000
(0.0000)
-0.1347
(0.0246)
0.066
0.0258
(0.0120)
0.382
0.391
0.000
0.031
IND_SCOPE2
FUNC_SCOPE
2
0.1203
(0.0641)
0.060
0.1043
(0.0631)
0.098
-0.0242
(0.0715)
-0.0099
(0.0060)
0.0346
(0.0128)
0.1039
(0.0639)
0.097
0.007
0.104
0.0345
(0.0139)
-0.0002
(0.0002)
0.1063
(0.0638)
0.277
0.096
FUNC_SCOPE
0.2078
(0.1025)
-0.0509
(0.0405)
0.043
0.209
FIXED EFFECTS
THER. AREA
Inc.
Inc.
Inc.
Inc.
Inc.
PHASE
Inc.
Inc.
Inc.
Inc.
Inc.
FIRM
Inc.
Inc.
Inc.
Inc.
Inc.
N
129
129
129
129
129
Log likelihood
LR Test (#vars)
LR Test (R-U)
-465.28
84.04
-465.12
0.000
-461.99
-464.95
-464.85
84.26
0.000
90.63
0.000
84.70
0.000
84.89
0.000
0.31
0.576
6.58
0.010
0.66
0.418
0.85
0.356
35
Table 6: Clinical Phase Results
(DV = Drug development project completion TIME)
Variable
CONSTANT
(4)
Phase 1 & 2
β
P>|z|
(SE)
-5.7820
0.001
(1.8012)
(5)
Phase 3 & 4
β
P>|z|
(SE)
-2.6237
0.000
(0.4003)
0.0001
(0.0015)
0.0948
(0.0333)
-0.0518
(0.0293)
-0.0001
(0.0001)
0.0002
(0.0003)
0.0049
(0.0095)
CONTROL
SUBJECT
CENTERS
NOVELTY
0.939
0.004
0.078
0.339
0.481
0.604
EXPERIENCE
TA_EXP
IND_EXP
PHASE_EXP
0.0749
(0.0862)
0.0305
(0.2234)
-0.1081
(0.0445)
0.385
0.891
0.015
0.0402
(0.0240)
0.0193
(0.0549)
-0.0004
(0.0033)
0.094
0.725
0.899
ECONOMIES
SCALE
TA_SCOPE
IND_SCOPE
FUNC_SCOPE
-0.0003
(0.0004)
-0.0332
(0.0716)
0.0903
(0.0496)
-0.3195
(0.2799)
0.505
0.642
0.068
0.254
-0.0003
(0.0003)
-0.1314
(0.0262)
0.0247
(0.0133)
0.1756
(0.0824)
0.236
0.000
0.063
0.033
FIXED EFFECTS
THER. AREA
Inc.
Inc.
Not Inc.
Not Inc.
FIRM
Inc.
Inc.
N
33
96
PHASE
Log likelihood
-59.46
LR Test (#vars)
54.11
-314.01
0.000
74.95
0.000
36
Table 7: Hazard Rate Multiplier*
Min.
Mean−SD
Mean
Mean+SD
Max
TA_EXP
0.00
Out of Sample
1.33
4.45
26.43
TA_SCOPE
-0.11
-2.04
-4.91
-7.78
-11.41
IND_SCOPE
1.02
Out of Sample
17.07
35.75
75.69
FUNC_SCOPE
0.00
0.02
0.02
1.73
3.43
* Calculated using statistically significant coefficient estimates of Model 4 in Table 4.
37