Regulatory
Is Risk-based Monitoring an Appropriate Methodology
for Clinical Trials in Emerging Regions?
An Analysis of Clinical Site Performance Across Global Geographic Regions
The current industry climate demands that sponsors of
clinical trials find ways to reduce clinical trial complexity
and drug development costs while getting more value
from limited research and development budgets. Today,
more than 30 per cent of a clinical trial budget is allocated
to cover site monitoring costs, and more than 50 per cent
of that is spent on source document verification (SDV)
activities. 1 While SDV has traditionally been done on 100
per cent of data points collected in a clinical trial, research
shows that less than 3 per cent of all case report form
(CRF) data is actually changed due to SDV activities (i.e.,
post-data capture monitoring and data cleaning) 2. Given
the high resource demand and cost of the traditional 100
per cent SDV approach, and the mounting evidence that
proves it has a minimal impact on data quality, many life
science organisations are applying risk-based monitoring
methodologies in line with regulatory guidance from the
FDA and EMA.
Risk-based monitoring allows life sciences companies
to more effectively target and prioritise resources
around identifiable risks that might compromise the
quality and integrity of clinical trial data. Implementing
this methodology, often described by activities such as
reduced SDV or targeted monitoring, requires a thorough
assessment of risks prior to study start as well as their
documentation in operational quality management plans
(e.g., site monitoring and data management plans).
Monitoring strategies should be tailored based on studyspecific risks. While risk categories such as critical data,
the type of patient and the investigational product can
be easily identified up front, it is often challenging to
determine the quality and experience level of investigative
sites participating in a study.
When it comes to emerging regions—where sites and
monitors generally have less experience in clinical research
than those in North America and Western Europe—
the conventional wisdom is that study teams need to
apply more rigorous on-site monitoring, as the level of
compliance and quality is expected to be problematic.
That assumption prompted us to ask whether sites in
emerging regions are indeed showing signs of lower
quality that warrant more intensive scrutiny in a riskbased monitoring paradigm.
To help answer this question, we analysed several key
risk indicators related to site quality and compared them
across global regions. We computed an aggregate value
for each comprising data from more than 6500 studies
contributed from over 120 sponsor organisations. 3 While
we expected to see indications of lower site quality and
26 Journal for Clinical Studies
lower patient enrolment, our analysis showed that sites in
emerging regions are actually trending more favourably
than those in established regions!
Methodology and Analysis
The metrics we assessed by global region include the
following:
•
•
•
•
Subject enrolment per site
eCRF auto-query rate
Subject visit to eCRF entry cycle time
Data management query opened-to-answered
cycle time.
For each metric above, we computed an aggregate
value for each of eight global geographic regions. They
include: North America, Western Europe, Eastern Europe,
Pacific Asia, South America, Central America, Middle
East and Africa. North America and Western Europe, for
the purpose of this analysis, are considered “established
regions,” while the other six regions are considered
“emerging regions.”
Figures 1 through 4 present the results for each metric,
and what they reveal is quite striking. Starting with subject
enrolment (Figure 1), five of the six emerging regions
have demonstrably higher per-site enrolment rates than
the established regions. And the one emerging market
that is not higher—the Middle East—is at least on a par
with North America at ~5.3 subjects per site, and not far
behind Western Europe (5.8). Enrolment performance
is particularly impressive in South and Central America,
both with rates of over 8 subjects per site. The excellent
per-site enrolment performance in these two regions is
especially interesting to note given that previous research
has shown that these regions contributed less than three
per cent of the total research subjects recruited globally. 4
This may be indicative of longer regulatory approval
periods, which often results in rapid enrolment at a few
sites later in the enrolment period.
eCRF auto-query rate is an important indicator of
eCRF data quality. In particular, a higher auto-query
rate indicates that a site is making a relatively high
number of mistakes when transcribing data from patient
source documents into the sponsor-provided eCRF.
The reasons for data entry errors are variable, but may
include insufficient training of site staff and/or lack of
engagement and attention to detail by those staff. Figure
2 presents auto-query rates and reveals that overall there
is no significant degradation in eCRF data quality in the
emerging regions as compared to the established ones.
Volume 6 Issue 2
Regulatory
at least as well as the two established ones. And again,
as with auto-query rates, sites in Eastern Europe and
Pacific Asia are performing significantly better than sites
in North America and Western Europe with respect to
timely entry of subject visit data. Both of the established
regions are averaging over 9 days (9.3 and 9.4), while the
two emerging regions are averaging less than 8 days (6.9
and 7.7).
Subject Enrollment Rate
(Subjects per Site)
9
7
5
3
1
North
America
Western
Europe
Eastern
Europe
Pacific
Asia
South
America
Central
America
Middle
East
Africa
Figure 1. Subject Enrolment per Study at Sites in
Different Geographic Regions
Source: Medidata Insights™ metrics warehouse
In fact, the rates in Eastern Europe and Pacific Asia are
significantly lower than in North America and Western
Europe. This is particularly impressive given that the
sites in these two emerging regions are managing more
subjects per study on average than in the two established
regions (as shown in Figure 1). One might expect that
managing more subjects in a study would present
greater opportunity for errors during eCRF data capture;
however, this factor may be offset to the extent that sites
in established regions are conducting more studies on
average than sites in emerging regions.
eCRF Auto-Query Rate
(Queries per 1,000 Datapoints)
60
50
40
Figure 3. Average Cycle Time from Subject Visit to
eCRF Entry at Sites in Different Geographic Regions
Source: Medidata Insights™ metrics warehouse
Data management query opened-to-answered cycle
time is another important measure associated with both
eCRF data quality and site responsiveness. As shown in
Figure 4, we once again see no particular issues with this
cycle time at sites in emerging regions as compared with
sites in established regions. Sites in Eastern Europe and
Pacific Asia appear to be on a par with sites in North
America and Western Europe, while sites in South and
Central America are performing significantly better than
in the other regions.
30
Data Mgt. Query Opened-to-Answered
Cycle Time (Days)
20
12
10
10
North
America
Western
Europe
Eastern
Europe
Pacific
Asia
South
America
Central
America
Middle
East
Africa
Figure 2. Rate of Auto-Queries per Volume of eCRF Data
Entered at Sites in Different Geographic Regions
Source: Medidata Insights™ metrics warehouse
Subject visit to eCRF entry cycle time is also a critical
measure of site quality, indicating not only eCRF data
quality but site responsiveness and overall engagement in
the study as well. This cycle time is of particular urgency
for successful risk-based monitoring, as long delays
in receiving subject data and information from sites
confounds a study team’s ability to proactively identify
areas of emerging risk. The critical importance of timely
data entry by sites to support effective centralised and
remote monitoring was aptly noted in TransCelerate’s
position paper on risk-based monitoring issued in 2013. 5
The results for this cycle time—shown in Figure 3—reveal
again that sites in the emerging regions are performing
www.jforcs.com
8
6
4
2
North
America
Western
Europe
Eastern
Europe
Pacific
Asia
South
America
Central
America
Middle
East
Africa
Figure 4. Average Cycle Time from Data Management
Query Opened to Query Answered at Sites in Different
Geographic Regions
Source: Medidata Insights™ metrics warehouse
Results and Conclusion
So what are we to make of these results? While the metrics
we selected for this analysis do not cover all aspects of
site conduct and quality, these measures do allow us to
make the following key observations:
Journal for Clinical Studies 27
Regulatory
1. Enrolment: Sites in emerging regions tend to enroll
more subjects per study than sites in established
regions, but this may be due to the volume of
competing studies in North America and Western
Europe.
2. Data Quality: Sites in emerging regions tend to
have less—and certainly no more—difficulty with
reliable eCRF data capture than sites in established
regions.
3. Responsiveness: Sites in emerging regions tend
to be more timely with data capture and cleaning
than sites in established regions. This again may
be reflective of the workload burden of site staff
in established regions where multiple competing
trials exist.
The above observations are supported consistently
and clearly across all four metrics, and the volume of
studies and sites contributing to these metrics across
all regions is significant; therefore there is little doubt
that the results are meaningful. The outcome may at first
appear to be counter-intuitive because many experts
have expressed concern that less experienced sites—
especially in these emerging regions—will struggle with
good clinical practice (GCP) compliance and high-quality
conduct of clinical research. While this concern may
yet be warranted for some aspects of site conduct, the
findings from our analysis suggest that sites in emerging
regions are highly motivated to participate in research.
They show themselves to be engaged and responsive,
at least with respect to subject recruitment and patient
data capture and management. The motivation may be
driven by the financial opportunity, relative to clinics
in Western Europe and North America that often face
challenges associated with low financial return on their
clinical research efforts (when compared with their nonresearch practices).
Whatever the cause, these results actually bode
well for moving more confidently toward a risk-based
monitoring paradigm across the globe. What is most
critical in this paradigm is to conduct an effective risk
assessment prior to study start, including site-specific
risks. And appropriate levels of scrutiny should be
planned across all sites in a global study, with processes
in place to remediate emerging site quality issues as they
are identified. However, as these results suggest, study
teams should not automatically assign higher risk to
sites in non-established regions, at least when it comes
to the risk-categories measured here. This also means
that monitoring plans should not necessarily apply more
intensive on-site monitoring—including more frequent
visits and higher levels of SDV—to sites simply based on
their geography.
An exciting implication here is that the resource
efficiencies and higher quality promised through riskbased monitoring need not be restricted to studies
conducted in North America and Western Europe. It is
time to move forward with risk-based monitoring globally!
28 Journal for Clinical Studies
References
1. IOM (Institute of Medicine). 2010. Transforming
Clinical Research in the United States: Challenges
and
Opportunities:
Workshop
Summary.
Washington, DC: The National Academies Press.
2. Denise Myshko, “Risk-Based Monitoring: A New
Way to Ensure Quality Data,” PharmaVOICE
(January 2014): 24.
3. Medidata Insights™ metric warehouse, 2014
4. Medidata
Solutions,
“Oncology
Subject
Recruitment Trending Down in All Geographies
Except North America.” Applied Clincial Trials. Aug
26, 2013.
5. Position
Paper:
Risk-Based
Monitoring
Methodology. TransCelerate BioPharma Inc. 2013.
Stephen Young brings more than 20 years of
experience to his role as principal engagement
consultant at Medidata Solutions. Since
joining the company in 2010, he has led the
development of Medidata’s insights and
site monitoring solution portfolios, providing
customers with turnkey metrics and analyticsdriven solutions to improve site monitoring
workflows. Previously, he headed eClinical
services for J&J Global Clinical Operations. He
holds a BS in mathematics from St. Joseph’s University and an MS
in mathematics from Villanova University.
Email: [email protected]
Kyle Given is principal of consulting services
at Medidata Solutions, focusing on risk-based
monitoring. With over 20 years of industry
experience, Kyle has spent the last 10+ years
holding strategic leadership positions in
business operations, where he has helped
design and deploy large global clinical
development teams and developed skills in
technology enablement, resourcing models,
process improvements, business analytics and
training. He received his BA from Franklin & Marshall College and
holds a graduate degree from Susquehanna University.
Email: [email protected]
Laurie Falkin is a director at Medidata Solutions,
where she leads initiatives to support the use of
risk-based monitoring technologies at clinical
development organizations. She has more than
10 years of experience in the healthcare and life
science industries. Laurie holds an MBA from
New York University Stern School of Business
and received her BS in microbiology from The
University of Massachusetts—Amherst.
Email: [email protected]
Volume 6 Issue 2